Commit Graph

389 Commits

Author SHA1 Message Date
Andrea Maria Piana 8931b14c4e
Explicitly init messenger (#1821)
Messenger needs to wait until all the topics/mailservers are loaded in
the client, so we explicitly add a way to call Init.
2020-01-29 20:40:06 +01:00
Pedro Pombeiro 741f43cb0c Fix wrong field being logged in `EnvelopesMonitor` 2020-01-23 17:19:10 +01:00
Andrea Maria Piana a4f88d0017
Fix ens names and contact (#1810)
Currently the wrong pk format was used to query the in-memory contacts,
this commit corrects the behavior.
2020-01-23 09:24:28 +01:00
Adam Babik bc2d018483
Add support for request messages by topics (#1805) 2020-01-21 08:11:24 +01:00
Adam Babik 79b8112f89
Split shhext into shhext and wakuext (#1803) 2020-01-20 21:56:06 +01:00
Andrea Maria Piana 456bcfa022
Peg clock value to whisper timestamp (#1804)
This commit pegs the clock value to maximum + 120 seconds from the whisper
timestamp.
In this way the we avoid the scenario where a client makes the timestamp
increase arbitrarely.
2020-01-20 17:44:32 +01:00
Pedro Pombeiro d4710faae2 In progress: Use Nimbus keystore 2020-01-20 13:15:17 +01:00
Pedro Pombeiro 6537cae606 Nimbus node support 2020-01-20 13:15:17 +01:00
Andrea Maria Piana 25d46c6d82
Fix waku tests & contact ens (#1802) 2020-01-17 13:39:09 +01:00
Pedro Pombeiro 7f45c43196 make lint-fix 2020-01-16 13:18:36 +01:00
Andrea Maria Piana c569d8a4ed
Sync installation messages & contact requests (#1791) 2020-01-15 08:25:09 +01:00
Adam Babik 44aa313981
Make shhext and protocol work with Waku (#1777)
This change makes shhext and protocol submodule work with Waku and Whisper.
2020-01-13 20:17:30 +01:00
Jakub Sokołowski 330d177de6 specify gomarkdown/markdown version to fix conflicts
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-01-11 19:41:24 +01:00
Andrea Maria Piana 88a1d0111e
Add Commands (#1731)
This commit adds handling of Transaction commands.
2020-01-10 19:59:01 +01:00
yenda d3de88f790 fix stickers persistence (#1779) 2020-01-10 16:12:09 +01:00
Adam Babik 37a508a97b
Integrate Waku service (#1759) 2020-01-08 12:12:23 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 3eaacaa2f4 Move keystore_geth.go to eth-node/bridge/geth 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 287e5cdf79 Abstract `accounts.Key` and geth `keystore` 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 41cba814c8 Run `make vendor` 2019-12-30 13:25:12 +01:00
Pedro Pombeiro f494576ca2 Run `gofmt -s -w` 2019-12-20 11:24:26 +01:00
Adam Babik ea0f12b920
Waku/0 service (#1715) 2019-12-20 10:40:50 +01:00
Pedro Pombeiro 8d8880dc62 Get rid of `types.Whisper.Poll` method 2019-12-19 14:11:48 +01:00
Pedro Pombeiro ef87c330ce Improve parameter naming in `Whisper.DeleteKeyPair` 2019-12-19 14:11:48 +01:00
Pedro Pombeiro 0cf24d2c3e Remove unnecessary call to `GetFilter` 2019-12-19 14:11:48 +01:00
Pedro Pombeiro a5cf17cf4b Improve logging format in `messageProcessor.handleMessages` 2019-12-19 14:11:48 +01:00
Pedro Pombeiro 7f11030896 Implement AddPeer/RemovePeer on eth-node 2019-12-19 14:11:48 +01:00
Pedro Pombeiro 41a6502340 Remove `SelectKeyPair` from `eth-nodes/types` 2019-12-19 14:11:48 +01:00
Adam Babik 4f45bceb9a
Drop peers when rate limit is exceeded (#1732) 2019-12-16 20:38:50 +01:00
acolytec3 2d2f2fbc51 Add ValidateMnemonic method (#1713) 2019-12-16 14:55:02 +01:00
Andrea Maria Piana baa0767c26
Handle membership update message
This commit does a few things:

1) Handle membership updates using protobuf and adds the relevant
endpoints.
2) Store in memory a map of chats + contacts for faster lookups, which
are then flushed to disk on each update
3) Validate incoming messages

Sorry for the large pr, but you know, v1 :)
2019-12-10 15:20:28 +01:00
Adam Babik 0c6a4e0e65
fix Whisper submodule version; bump go-ethereum version 2019-12-09 16:26:00 +01:00
Adam Babik 6284d45b69
fix docker image build 2019-12-09 16:04:32 +01:00
Adam Babik a636f33109
Set Whisper version submodule to v6 (#1725) 2019-12-09 11:36:14 +01:00
Adam Babik 4ac4a61e20
Move whisper to status-go monorepo (#1720) 2019-12-09 11:06:04 +01:00
Andrea Maria Piana fd49b0140e
Move to protobuf for Message type (#1706)
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`

* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.

* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.

* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
2019-12-05 17:25:34 +01:00
Jakub Sokołowski f4d4c66d5b upgrade whisper: 1.6.1 > 1.6.2
This adds new Prometheus metrics

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-12-02 13:17:52 +01:00
Pedro Pombeiro 26b6d7c36a Create a home submodule for Eth node bridges- Rename StatusBackend to GethStatusBackend 2019-11-27 17:02:09 +01:00
Adam Babik 47ccf32cdb
Upgrade zap to v1.13.0 (#1694) 2019-11-25 21:16:00 +01:00
Adam Babik 10635d555f
Make extkeys as a submodule (#1691) 2019-11-22 13:58:03 +01:00
Adam Babik e1a8ba1ba5
Fix modvendor (#1690) 2019-11-22 13:24:20 +01:00
Pedro Pombeiro ed5a5c154d
Move to monorepo structure (#1684)
Move to a monorepo structure with submodules

- Rename status-protocol-go to status-go/protocol
2019-11-21 17:19:22 +01:00
Pedro Pombeiro 2dd74da23d Integrate Nimbus status-protocol-go 2019-11-18 14:30:03 +01:00
Adam Babik c199c8f342
Update Whisper and introduce rate limiting (#1673) 2019-11-18 11:22:23 +01:00
Pedro Pombeiro 6ce437255e Remove FCM implementation 2019-11-17 17:54:55 +01:00
Jakub Sokołowski 4c313c7032 add tcp-pinger for measuring rtt of mailservers
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-11-15 13:18:08 +01:00
Andrea Maria Piana 9d7c570593
Add PrepareContent and upgrade status-go (#1674) 2019-11-15 09:52:28 +01:00
Andrea Maria Piana 89659f85b4
Upgrade status-protocol-go (#1664) 2019-11-08 08:36:12 +01:00
Andrea Maria Piana a659685e3f
Add parsed message (#1660)
This commits add a field (parsedMessage) to the json payload sent to
status-react.
This field is the parsed version of the transit message.
The code is all in dedup, I will re-organize it once we made all the
necesseary changes.
2019-11-06 17:23:11 +01:00
Adam Babik 183d626256
Implement mailserver Deliver() with MessagesRequest (#1653)
The main difference is that it takes `whisper.MessagesRequest` as an argument instead of `whisper.Envelope`.
2019-11-04 21:15:21 +01:00
Jakub Sokołowski c1f17cb4a3 use prometheus/client_golang for metrics
also re-implement mailserver metrics

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-11-04 13:18:50 +01:00
Andrea Maria Piana 9f2a4b378f
Verify ens endpoint, only against mainnet (#1657)
* String payload

* Verify ENS Names

* Update status-protocol-go
2019-11-04 11:08:22 +01:00
Adam Babik a244b01a26
Upgrade migrate (#1643) 2019-10-14 16:10:48 +02:00
Pedro Pombeiro c874960215
Use status-protocol-go/bridge/geth (#1638)
* Use status-protocol-go/bridge/geth
2019-10-14 09:53:38 +02:00
Pedro Pombeiro d87caf57e3
Add dependabot config file (#1636)
* Add dependabot config file

* Fix status-protocol-go dependency
2019-10-11 07:00:46 +02:00
Jakub Sokołowski 294399916e add script for easier setup of mailserver
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-10-10 11:27:32 +02:00
Pedro Pombeiro a2f106e4c5
Update for status-protocol-go API changes (#1627) 2019-10-09 16:22:53 +02:00
Adam Babik 26880b83d7
Upgrade geth to 1.9.5 and Whisper (#1617) 2019-10-04 17:21:24 +02:00
Andrea Maria Piana 40e66e6186
upgrade status-protocol-go (#1619) 2019-09-26 11:26:33 +02:00
Andrea Maria Piana 9a3ed980c9
Generate gfycat and identicon in status-protocol-go (#1611) 2019-09-26 09:01:17 +02:00
Andrea Maria Piana 1a47893e75
Upgrade status-protocol-go (#1593) 2019-09-02 11:29:06 +02:00
Andrea Maria Piana b8ea79a3f0
Upgrade status-protocol-go (#1586) 2019-08-29 08:33:46 +02:00
Andrea Maria Piana e118e89107
Add persistent datasync, use single database and handle negotiated filters (#1580) 2019-08-27 14:04:15 +02:00
Andrea Maria Piana 06dc227071
Add Metadata to messages, expose new messenger methods (#1571)
* Add Metadata to messages, expose new messenger methods

This commits modifies deduplicator so that it takes a `StatusMessage`
instead of `WhisperMessage` and also returns a `Metadata` field which is
then passed back by the client when confirming messages, which fixes the
issue we had with not confirming pfs messages.
2019-08-20 13:20:25 +02:00
Adam Babik 8383feea04 add API to shhext to manage messages 2019-08-12 19:44:31 +02:00
Andrea Maria Piana 9ae7d2b6d5 Move envelopes tracking to status-protocol-go
This commit moves envelopes tracking to status-go.
Post endpoint is not going to track envelopes anymore, as that's taken
care on status-protocol-go side, so this is a breaking change, and
version is updated accordingly.
2019-08-12 19:44:31 +02:00
Andrea Maria Piana a3a413b5d9 Add SaveChat, Chats, DeleteChat, SaveConctat & Contacts endpoint 2019-08-05 16:03:18 +02:00
Adam Babik ce7b0fc376 conslidate protocol databases 2019-08-01 10:40:54 +02:00
Andrea Maria Piana 9de77b21b2 Add datasync, v1messages & disable discovery topic options
Adds support for datasync, V1Messages and disabling the discovery topic.
This is a backward compatible change as long as they are not toggled on
(they are not by default).
2019-07-29 19:39:44 +02:00
Andrea Franz dcb0fa5262
add multi-account support (#1500)
* add multiaccount support

add multi account ImportPrivateKey and StoreAccount

test derivation from normal keys

* add multiaccount to mobile pkg

* use multiaccount params structs from the mobile pkg

* move multiaccount tests together with the other lib tests

* fix codeclimate warning and temporarily increase methods threshold

* split library_test_utils.go to avoid linter warnings
2019-07-24 20:59:15 +02:00
Adam Babik e93d994460 integrate status-protocol-go 2019-07-24 18:46:43 +02:00
Adam Babik 664488faf9
add an adapter for zap and geth loggers (#1534) 2019-07-24 09:58:01 +02:00
Adam Babik 1ab2e88bf5
Sync messages in a loop until error or no cursor (#1502) 2019-06-26 18:17:41 +02:00
Andrea Maria Piana f6fba1d3d6 Publish contact code periodically 2019-06-19 17:38:45 +02:00
Andrea Maria Piana 1aa3e2812a Add support for partitioned topic 2019-06-19 17:38:45 +02:00
Adam Babik c9e99c432d migrate to go 1.12 and go modules 2019-06-12 13:12:00 +02:00
Andrea Maria Piana 4ab08629f6 Add postgres
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
2019-05-15 11:01:34 +02:00
Andrea Maria Piana 061f10e58d
Publish rlp.RawValue instead of envelope (#1459)
As part of a performance profiling of mailserver we noticed that most of
the resources on a query are spend decoding the whisper envelope.
This PR changes the way we store envelopes encoding the Topic into the
database key, so we can check that and we are able to publish the
envelope rawValue if it matches.
The change is backward compatible as only newly added envelopes will
have the new key, while old ones will have to be unmarshaled.
2019-05-09 12:58:02 +02:00
Andrea Maria Piana 354e6981ba
Add network incentivisation service (#1452) 2019-04-29 14:05:49 +02:00
Dmitry Shulyak 866342d5e3
Update whisper to v1.4.12 (#1447) 2019-04-23 20:16:48 +03:00
Dmitry Shulyak a904d9325e
Notify users that envelope was discarded and retry sending it (#1424)
* Notify users that envelope was discarded and retry sending it

* Update Gopkg files with released whisper version

* Forgot to remove signal after refactoring
2019-04-02 13:40:45 +03:00
Dmitry af403280a6 Upgrade geth to 1.8.23 (includes necessary whisper update) 2019-03-14 14:39:31 +02:00
Dmitry b2f02d0dbe Use go-ethereum with fixed logger 2019-03-07 16:26:20 +02:00
Dmitry Shulyak cf8f20b23c Use lumberjack library to rotate logs and add config that enables it (#1399) 2019-03-01 14:37:13 +01:00
Dmitry 5d829f89f8 Pull all recent changes to 1.8.21 tag 2019-02-25 09:13:52 +02:00
Dmitry Shulyak 224bc7857c Mark envelope as confirmed when it was received from a mail server (#1378) 2019-02-20 18:10:59 +01:00
Andrea Maria Piana 81d8ca82a2
Add confirm messages processed by ID (#1375)
Currently PFS messages are decrypted and therefore modified before being
passed to the client. This make IDs computation difficult, as we pass
the whole object to the client and expect the object be passed back once
confirmed.
This changes the behavior allowing confirmation by ID, which is passed
to the client instead of the raw object.
This is a breaking change, but status-react is already forward
compatible.
2019-02-19 13:58:42 +01:00
Pedro Pombeiro 55b3f31318 Add `SendDataNotification` method (#1352) 2019-01-18 14:03:32 +01:00
Dmitry aa09d854ca Prevent frequent requests with same topics 2019-01-18 14:52:33 +02:00
Adam Babik fd48ad8b29
geth: upgrade geth to 1.8.21 (#1350) 2019-01-16 15:42:00 +01:00
Andrea Maria Piana 61bbbdde58
Revert "Change `NotifyUsers` to allow only data JSON element. Part of status-im/status-react#6772" (#1347)
This reverts commit cddf2e1c6c.
2019-01-14 13:09:51 +01:00
Dmitry Shulyak f151be54c6
Close both sides of the stream when client is done with request (#1313)
Libp2p keeps stream open if EOF wasn't seen and we called Close method.
The most important change is that reader now uses FullClose util, that will
wait for EOF character before closing the stream.
2019-01-02 09:45:52 +02:00
Adam Babik b1f9030177
update to geth v1.8.20 (#1327) 2018-12-19 11:02:07 +01:00
Adam Babik 74cb16c456
add SyncMessages method to shhext api (#1309) 2018-12-14 12:21:34 +01:00
Andrea Franz fdcd20e14d
add missing patch file (#1319) 2018-12-12 16:09:39 +01:00
Andrea Franz e1c258d041
update go-ethereum with patch 0044-import-extended-key-by-purpose (#1318) 2018-12-12 11:13:10 +01:00
Adam Babik db691fd763
fix golang/mock (#1310) 2018-12-10 13:53:27 +01:00
Adam Babik 913dbfca9b
Support Mail Server data synchronization (#1302)
These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
2018-12-06 10:48:28 +01:00
Dmitry Shulyak a609b468fe
Mail peer store and connection manager (#1295)
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history

When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.

If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.

If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.

We will have two components that will rely on this logic:
1. requesting history

If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.

2. confirmation from selected mail server

Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.

I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server

Currently this feature is hidden, as certain changes will be necessary in status-react. 

partially implements: https://github.com/status-im/status-go/issues/1285
2018-12-05 15:57:05 +02:00
Pedro Pombeiro cddf2e1c6c Change `NotifyUsers` to allow only data JSON element. Part of status-im/status-react#6772 2018-11-30 13:26:27 +01:00
Dmitry Shulyak aac706fe4c
Use confirmation that bundle is delivered for sending feedback to user (#1284)
* Pull whisper confirmations changes

* Use batch confirmations as a signal that envelope was sent into the network
2018-11-27 08:30:15 +02:00
Adam Babik 52a1bdfed6
Upgrade geth 1.8.17 plus add metrics during compilation time (#1273)
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.

The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
2018-11-14 08:03:58 +01:00
Andrea Maria Piana ee3c05c79b
Change handling of skipped/deleted keys & add version (#1261)
- Skipped keys

The purpose of limiting the number of skipped keys generated is to avoid a dos
attack whereby an attacker would send a large N, forcing the device to
compute all the keys between currentN..N .

Previously the logic for handling skipped keys was:

- If in the current receiving chain there are more than maxSkip keys,
throw an error

This is problematic as in long-lived session dropped/unreceived messages starts
piling up, eventually reaching the threshold (1000 dropped/unreceived
messages).

This logic has been changed to be more inline with signals spec, and now
it is:

- If N is > currentN + maxSkip, throw an error

The purpose of limiting the number of skipped keys stored is to avoid a dos
attack whereby an attacker would force us to store a large number of
keys, filling up our storage.

Previously the logic for handling old keys was:

- Once you have maxKeep ratchet steps, delete any key from
currentRatchet - maxKeep.

This, in combination with the maxSkip implementation, capped the number of stored keys to
maxSkip * maxKeep.

The logic has been changed to:

- Keep a maximum of MaxMessageKeysPerSession

and additionally we delete any key that has a sequence number <
currentSeqNum - maxKeep

- Version

We check now the version of the bundle so that when we get a bundle from
the same installationID with a higher version, we mark the previous
bundle as expired and use the new bundle the next time a message is sent
2018-11-05 20:00:04 +01:00
Igor Mandrigin ba504e99c4
Mailserver: return error response. (#1244) 2018-10-18 12:25:00 +02:00
b00ris 820217ea7a added ulc patch (#1239) 2018-10-16 14:27:11 +03:00
Igor Mandrigin 1136176f4a Make statusd compile with go 1.11 (#1225)
* upgrading github.com/rjeczalik/notify to the latest release
2018-10-04 17:57:39 +02:00
Dmitry 5b551c67fa Use status-im whisper/whisperv6 module everywhere 2018-10-03 09:27:51 +03:00
Adam Babik ac8da3c0f0
Use patched go-ethereum from our fork instead of patching it on-the-fly (#1184)
* update to 1.8.16
2018-09-27 21:16:15 +02:00
Dmitry Shulyak 529e18af33
Update geth to v1.8.15 (#1213)
* Update geth to v1.8.15

* Apply patches
2018-09-25 11:55:54 +03:00
Andrea Maria Piana dcaf8caed0
Add x3dh key exchange (#1127)
* Add x3dh key exchange

* Encrypt using the double ratchet

* Multi device with auto-pairing

* Add pfs enabled flag
2018-09-24 20:07:34 +02:00
Dmitry aab84e53dc Ephemeral rendezvous client generates key using seckp256k1 curve 2018-09-24 13:37:06 +03:00
Dmitry 6f937d4ddf Update rendezvous client to a latest version 2018-09-13 09:50:38 +03:00
Adam Babik bf1d1ee0ea
Remove EnvelopeTranced and add Whisper metrics as a patch (#1177) 2018-08-31 10:09:38 +02:00
Adam Babik a571eeb7a4
add patch 0036-fix-whisper-expire-loop (#1175) 2018-08-28 10:49:08 +02:00
Dmitry Shulyak f150d678de
Update geth to 1.8.14 (#1171)
* Update to geth v1.8.14

* Remove patches that were merged upstream

* Apply patches before 0016

* Fix 0016 and apply it

* Apply everything else

* Pass gas limit as a second argument to simulated backend
2018-08-27 11:22:21 +03:00
Dmitry Shulyak 2cfcde0283
Remove patches to discovery v5 (#1144) 2018-08-24 11:24:22 +03:00
Dmitry Shulyak d8b5ba7fbf
Add generated cht for ropsten and mainnet (#1166) 2018-08-24 06:05:06 +03:00
Dmitry bfdf823628 Add delay to refresh cycle with patch 2018-08-15 04:25:18 +03:00
Dmitry 6c6fb4d0b3 Remove patch 0031 2018-08-15 04:25:18 +03:00
Adam Babik ecc39735f5
Upgrade geth to 1.8.13 (#1140) 2018-08-07 15:31:06 +02:00
Dmitry eeca435064 Add rendezvous implementation for discovery interface
Update vendor

Integrate rendezvous into status node

Add a test with failover using rendezvous

Use multiple servers in client

Use discovery V5 by default and test that node can be started with rendezvous discovet

Fix linter

Update rendezvous client to one with instrumented stream

Address feedback

Fix test with updated topic limits

Apply several suggestions

Change log to debug for request errors because we continue execution

Remove web3js after rebase

Update rendezvous package
2018-07-25 15:10:57 +03:00
Ivan Daniluk 801053a17d
Remove jail package (#1106) 2018-07-24 12:05:21 +02:00
Ivan Daniluk 93210061ad
Add goroutines count to geth metrics (#1077) 2018-07-03 20:15:16 +02:00
Adrià Cidre f19e3eae8e
Move TestSentEnvelope to status-scale repo (#1071) 2018-07-02 22:29:03 +02:00
Andrea Franz 809db97e54
mailserver pagination (#1039)
* mailserver sends envelopes in descending order

* add limit value in mailserver request payload

* mailserver sends messages up to the limit specified in the request

* update Archive method to return key and error

* processRequest returns the next page cursor

* add cursor to mailserver request

* add limit and cursor to request payload

* fix request limit encoding

* wait for request completed event in TrackerSuite/TestRequestCompleted

* add cursor to mailserver response

* fix cursor position in payload

* add e2e test for mail server pagination

* validate mail server response size

* remove old limitReached var

* fix lint warnings

* add whisper patch

* fix tests after rebase

* check all return values to avoid lint warnings

* check that all messages have been retrieved after 2 paginated requests

* fix lint warnings

* rename geth patch

* merge mailserver patches into one

* add last envelope hash to mailserver response and EventEnvelopeAvailable event

* update whisper patch

* add docs to MailServerResponse

* update whisper patch

* fix tests and lint warnings

* send mailserver response data on EventMailServerRequestCompleted signal

* update tracker tests

* optimise pagination test waiting for mailserver to archive only before requesting

* rollback mailserver interface changes

* refactoring and docs changes

* fix payload size check to determine if a limit is specified

* add more docs to the processRequest method

* add constants for request payload field lengths

* add const noLimits to specify that limit=0 means no limits
2018-07-02 09:38:10 +02:00
Andrea Franz b913c98573
remove formatting from metrics.ResettingTimer when requested in raw format (#1052) 2018-06-27 11:46:50 +02:00
Pedro Pombeiro 3b052098fe Update to geth 1.8.11. Closes #1033
- Replace deprecated common.Hex with hexutil.Encode.
- Remove upstreamed 0010-geth-17-fix-npe-in-filter-system.patch.
- Remove upstreamed 0020-discv5-metrics.patch.
- Remove upstreamed 0026-ethdb-error-deadlock.patch.
- Update goleveldb to same version used by geth 1.8.11.
- Update PublicTransactionPoolAPI.GasPrice return type to match type in internal geth interface.
2018-06-22 18:04:13 +02:00
Adrià Cidre 51ab9053ab
[#1040] Patch go-ethereum reducing from Warn to Info "message loop" logs (#1048) 2018-06-22 11:57:52 +02:00
Andrea Franz fa390a52ae
RequestHistoricMessages response (#1009)
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Adrià Cidre 8141657ad7
[#1018] Add a flag to disable shell colors and move verbose logs to debug level (#1020) 2018-06-12 09:49:38 +02:00
Pedro Pombeiro 3f1b2fa57f Update go-duktape.v3 revision to include submitted patch
- Gets rid of noisy build warnings
2018-06-11 15:53:56 +02:00
Adam Babik b5f05b1ab5
Fix flaky TestSingleTopicDiscoveryWithFailover and others (#992)
Other changes:
* needed to patch that loop implementation in Discover V5 implementation in go-ethereum,
* fixed TestStatusNodeReconnectStaticPeers,
* fixed TestBackendAccountsConcurrently.
2018-05-26 09:37:13 +02:00
Dmitry Shulyak ed1cdf0418
Custom status bootnodes (#968)
This change makes invalidation mechanism more aggressive. With a primary goal to invalidate short living nodes faster. In current setup any node that became known in terms of discovery will stay in this state until it will fail to respond to 5 queries. Removing them earlier from a table allows to reduce latency for finding required nodes.

The second change, one adds a version for discovery, separates status dht from ethereum dht.
After we rolled out discovery it became obvious that our boot nodes became spammed with irrelevant nodes. And this made discovery process very long, for example with separate dht discovery takes ~2s, with mutual dht - it can take 1m-10m and there is still no guarantee to find a max amount of peers, cause status nodes is a very small part of whole ethereum infra.

In my understanding, we don't need to be a part of ethereum dht, and lower latency is way more important for us.

Closes: #941
Partially closes: #960 (960 requires futher investigations on devices)
2018-05-18 16:43:07 +03:00
Adam Babik 6da469140f
Refactor in-app metrics (#963)
Remove Prometheus and other metric types and use metrics package from go-ethereum.
2018-05-16 17:36:59 +02:00
Dmitry Shulyak ca962ddaa3 Whisper broadcast loop spams way too hard with debug level 2018-05-16 17:54:40 +03:00
Dmitry Shulyak 5aae87aba8
Prevent sending messages to flaky peers (#917) 2018-05-15 20:08:31 +03:00
Adrià Cidre 08b4d515c6
[#ISSUE-937] : Move mail server implementation to status-go repo (#942)
* [#ISSUE-937] : Move mail server implementation to status-go repo

* Applying dep-ensure
2018-05-11 21:43:07 +02:00
Dmitry Shulyak 7aa508765e Set timesource from config (#915) 2018-05-08 13:38:54 +03:00
Dmitry Shulyak 84cb5ca917 Ensure that wg.Add is called before wg.Wait
Now if Add is to be called it will be called before Wait, this is achieved
by doing following:
- if cancel() gets lock first and closes channelCh before spawnSync is called
we will exit right away
- if not than we will ensure that we hold a lock until syncers are spawned
so that cancel() will be blocked for this time and it will prevent whole Terminate() from
progressing
2018-05-04 10:28:15 +02:00
Dmitry Shulyak 707221954f
Make whisper tolerant to local time skews (#864)
This change adds adds an ability to use different source of time for whisper:

when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.

So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.

In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.

Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.

Closes: #687
2018-05-04 11:23:38 +03:00
Pedro Pombeiro acbf251a3c Fix race condition in `LDBDatabase.Close()`
- https://jenkins.status.im/job/status-go/job/race-check/54/consoleFull
2018-05-03 20:39:42 +03:00
Frank Mueller 123382cdf1
LevelDB vendor has now the same version as used in go-ethereum 1.8.5 (#883)
* LevelDB version is now same as in go-ethereum

* make dep-ensure untracked LevelDB storage.go
2018-05-01 19:09:19 +02:00
Pedro Pombeiro 7074048fa5 Add patch to fix race condition in `Downloader.Cancel`
- Submitted patch upstream in https://github.com/ethereum/go-ethereum/pull/16585
- Fixes Jenkins build in https://jenkins.status.im/job/status-go/job/race-check/48/
2018-05-01 16:29:27 +02:00
Frank Mueller 30cf19e0ae
Wrong command order (#879) 2018-04-30 23:36:28 +02:00
Igor Mandrigin f4cd8d27b5 Add `shhext_getNewFilterMessages` function to RPC APIs.
This function returns only the new messages from the filter, never
returns the same message for the same user twice.
2018-04-27 15:24:37 +02:00
Pedro Pombeiro a23b607597 Fix race condition in `0016-fix-leveldb-issue.patch`
https://jenkins.status.im/job/status-go/job/race-check/45/
2018-04-26 17:19:32 +02:00
Dmitry Shulyak b37fda7731
Update go-ethereum to 1.8.5 (#854)
* Rebase on 1.8.5

* Remove outdated patches and apply all others

* Use shh_post that returns hash

* Use bloom filter for request to mailserver

* Remove tests for sending messages without subbing first

* Fix deadlock in ethdb

* Expect null if receipt is not yet created

* Subscribe to messages before sending them in whisper test
2018-04-24 18:50:26 +03:00
Dmitry Shulyak 2f2dfe16c0
Add default peer limits configuration (#830)
* Add default peer limits configuration

If discovery is enabled for a given cluster - we will set a default
expected number of peers for each enabled service. For example:

- if cluster is rinkeby has a discovery enabled we will
  check which services are enabled
- if whisper is enabled we will set min and max limits by default
- if les is enabled and infura is not used we will set limits too

When statusd is used - configuration must be provided using configuration
supported by statusd.


* Fix deadlock in les peer set
2018-04-20 15:23:18 +03:00
Igor Mandrigin 4cc6028d59 Implement `personal_sign`.
This commit implements `personal_sign` RPC or web3 personal.sign
methods.

NB! Contains breaking API changes.
2018-04-17 11:18:58 +02:00
Adam Babik 0d652c3851
Expose only public APIs in inproc RPC client [breaking-change] (#815)
`CallRPC` binding, which is used as a provider for web3.js, exposes only public or whitelisted APIs.
2018-04-12 18:17:10 +02:00
Dmitry Shulyak ba9a25e284
Implement shh api extension that allows to confirm that message is sent (#814)
* Implement shh api extension that allows to confirm that message is sent

* Add a patch

* Fix linter

* Add readme

* Add tests for tracker

* Address review
2018-04-11 18:41:51 +03:00
Dmitry Shulyak b71b7eeadb Use web3code from separate repo and remove static 2018-04-09 15:42:36 +02:00
Dmitry Shulyak 38d7194a2a Use chain aware signer in simulated backend 2018-04-09 14:41:50 +03:00
Dmitry Shulyak 3f2e39d525 Update vendor because new geth modules were imported 2018-04-09 14:41:50 +03:00
Dmitry Shulyak 74ce515ab2
Add metrics to discv5 (#749) 2018-04-05 16:48:59 +03:00
Evgeny Danienko e2e2af73d7
isForwarded constants introduced 2018-03-27 18:22:29 +03:00
Evgeny Danienko b4c1cd7212
send in-house messages without subscription 2018-03-27 17:47:15 +03:00
Evgeny Danienko de0c03ca4e
patch sending message without subscription applied 2018-03-26 19:30:31 +03:00
Igor Mandrigin cfb3e6a080
Fix datarace when stopping a node. (#723) 2018-03-05 21:36:32 +01:00
Pedro Pombeiro 6cdea4ef97
Update codebase to leverage Whisper v6 (#703)
* Update project to use Whisper v6. Part of #638

* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.

* Enable light mode on Whisper v6 for non-mail servers. Part of #638

* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)

* Update bootnode addresses in staticnodes.json. Part of #638

* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638

* Move MakeTestNodeConfig to utils. Part of #638

* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
2018-03-02 10:25:30 +01:00
Dmitry Shulyak 988cd73217
Use newer CHT defined in go-ethereum (#711) 2018-03-01 15:23:41 +02:00
Pedro Pombeiro e4cbce12c4
Update to go-ethereum 1.8.1 (#702)
* Update `github.com/ethereum/go-ethereum` package to 1.8.1 branch. Part of #638
* Fix code due to some signature changes. Part of #638
* use upstream for whisper backend
* Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #638
* Take into account the DNS rebinding protection introduced in 1.8.0 by adding exception for localhost. Part of #638
* Add patches required for cross-compiled builds starting with geth 1.8.0. Only applied during build. Part of #638
* Update expected JSON result in `TestRegressionGetTransactionReceipt()` and `TestCallRawResultGetTransactionReceipt()`. Part of #665
* Fix some failing e2e tests. Part of #638
* Address comments in PR #702. Part of #638
2018-02-27 11:39:30 +01:00
Igor Mandrigin abb5df88d1
Wait for `Rollback` before closing the DB in the light client. (#696)
* Wait for `Rollback` before closing the DB in the light client.

* Unsubscribe from mux events properly, get rid of `time.Sleep`.
2018-02-23 11:49:57 +01:00
Igor Mandrigin c06d58addd Remove `notifications` package from go-ethereum. 2018-02-22 11:10:21 +01:00
Adam Babik 04d216dcc5 Remove non-go option (#682) 2018-02-21 15:37:46 +01:00
Dmitry Shulyak b2580c79d7
Add destructive tests (#646)
Network disconnect is introduced by removing default gateway, easily reversible condition.

On my local machine it takes 30 seconds for peers to reconnect after connectivity is restored. As you guess this is not an accident, and there is 30 seconds timeout for dial expiration. This dial expiration is used in p2p.Server to guarantee that peers are not dialed too often.

Additionally I added small script to Makefile to run such tests in docker environment, usage example:

```
make docker-test ARGS="./t/destructive/ -v -network=4"
```
2018-02-19 17:32:58 +02:00
Alex Kohler 87b12ddced Remove DefaultMinimumPow patches in favor of using whisper config (#683) 2018-02-19 15:53:40 +01:00
Dmitry Shulyak 4a49e840e3 Remove status backend from les internal api (#670) 2018-02-16 15:40:37 +01:00
Igor Mandrigin 004d3ef249 Remove unused `0007-README.patch`. 2018-02-15 13:08:32 +01:00
Igor Mandrigin 2e03cbcb5d Remove outdated `0003-dockerfiles-wnode-swarm.patch`. 2018-02-15 12:51:37 +01:00
Caner Çıdam 15c72f3c6b Apply geth patches on the fly #653 (#660) 2018-02-14 18:44:27 +01:00
Pedro Pombeiro 857b72e9fd
Create whisperv6 patch versions of `0004-whisper-notifications.patch` and `0009-whisper-envelopes-tracing.patch`. Closes #637 2018-02-12 16:44:04 +01:00
Pedro Pombeiro 9976018978
Create temporary whisperv6 patch file. Closes #636 2018-02-12 16:36:55 +01:00
Igor Mandrigin 9cc9982a6a
[#639] Disable filter removal in `go-ethereum`. (#650) 2018-02-12 13:02:25 +01:00
Dmitry Shulyak d0ef64a177 Maintain local copy of the nonce for each used address (#538) 2018-02-08 00:23:57 +02:00
Igor Mandrigin 156bbe1986 [#429] fix npe in geth's filter system. 2018-02-02 15:10:30 +02:00
Adam Babik 89d89681a3
Add support for metrics (#558)
It adds support for metrics (expvar and Prometheus) along with docker-compose files to run a Whisper test cluster.
2018-01-30 12:51:48 +01:00
Igor Mandrigin fb3d2ff6fe
Introduce the dependency vendoring tool: `dep`. (#551)
* Introduce `dep`, the dependency vendoring tool.

Use commits from `go-ethereum@release/1.7` for most of the dependencies.

* Update dependencies.
2018-01-25 14:08:43 +01:00
Adam Babik c153a60dc3
Clean up whisper log delivery (#555) 2018-01-19 15:53:16 +01:00
Dmitry Shulyak 0771e7d1b7 Use single codepath for sending transactions to a local and remote nodes (#527)
- new EthereumTransactor that provides higher level API for working with ethereum network, and it is fully conformant with ethclient
- new test rpc service that improves flexibility and coverage of txqueue manager tests
- run complete transaction sequantially for each address
- go-ethereum: Get pending nonce from transaction pool
- add a patch with getting nonce from transaction pool
2018-01-18 17:55:17 +01:00
Ivan Daniluk f93cd81d83
Upgrade to geth 1.7.3 and add geth patches (#492) 2018-01-15 21:26:41 +01:00
Gene 8acfc71043 Remove unused vendor libraries #198 (#480) 2017-11-30 13:36:20 +01:00
Ivan Daniluk 4cd362213f
Add env var to tune header writes delay (#464) 2017-11-22 21:13:11 +02:00
Ewetumo Alexander cb5ccb52c4 Emit messages logs for processing and sorting out messaging problems (#420)
* Add DeliveryService to Whisper to track incoming and outgoing messages.
* Have log tagged log lines for incoming and outgoing messages.
2017-11-06 14:10:52 -05:00
Adam Babik a19e502686 Remove go.rice in favour of go generate (#444)
As fetch.go was the only consumer of go.rice, it could be safely removed in favour of go-bindata and go generate.
Motivation to do that was the fact that go.rice throws panic in iOS when compiled with gomobile.
2017-11-03 18:11:37 -04:00
Adam Babik b9372459cc Rebase on geth 1.7.2 (#402) 2017-10-17 12:58:04 +03:00
Adam Babik 90acfedf7a Rebase geth 1.7.0 (#353) 2017-10-10 12:38:49 +03:00
Adam Babik 750612f2bc fix SendTransaction signature in go-ethereum (#355)
I created a separate method SendTransactionWithPassphrase which accepts passphrase as a second argument. It's an exact copy of SendTransaction except for calling wallet.SignTxWithPassphrase.
2017-09-22 18:16:22 +03:00
Oskar Thorén f88de68f86 Push notifications: ability to trigger notifications via public API (#327)
This PR provides a way for status-react to trigger push notifications to contacts whose FCMToken they possess. It thus solves the basic user story as outlined in #326
2017-09-15 18:57:34 +03:00
Ivan Daniluk 6a096607cf Add FetchAPI support and fix loop race [upd] #289 (#293)
This PR adds Fetch API and fixes #289 by using concurrency safe Otto VM wrapper wherever it's possible. This involves new package geth/jail/vm that is used by jail and by our forked ottoext/{fetch/timers/loop} packages.

It also adds more tests that are supposed to be run with --race flag of go test.
2017-09-08 14:55:17 +03:00
Adam Babik 39aeb3b09d vendor: clean up go-ethereum (#299) 2017-09-07 10:35:10 +02:00
Adam Babik 8fb2424ea5 Port the transactions queue from go-ethereum to status-go (#261)
Now all transactions in both cases are waiting for CompleteTransaction or DiscardTransaction to proceed independently from their destination: upstream of local
2017-09-04 15:56:58 +03:00
Oskar Thorén 680090bdf9 vendor: Upgrade go-stack 2017-09-03 11:15:03 +02:00
Ivan Daniluk 75398a19d5 Fix "Slow CreateAccount" #251 (#283) 2017-09-02 03:17:34 +07:00
Adam Babik 7433828a26 Validate JSON config (#240)
* exposed ValidateNodeConfig()
* extended NodeConfig interface with Validate() method
* added validate tag to some NodeConfig fields
2017-08-10 22:31:29 +07:00
Ivan Tomilov ebd77aabe2 Merging bug/whisper-on-geth1.6.1 (#236) which acts like develop
* static: updates Whisper test (to work with Geth 1.6.1)
* jail: VM persistence implemented
* jail: sendMessage/showSuggestions minor fixes (to be squashed)
* node: CHT and boot nodes auto-load implemented
* Replaced CHT data file from farazdagi's to tiabc's
* Rewrote config_test.go using testify having reduced it twice in size
* Increased SyncTime and panic timeout in tests
* Fixed test - remove go default test to testify/suite (#207)
* Add flag setup for RPCEnabled and add comment (#225)
* jail: register method handlers before running initial js in jail (#226)
* Console Jail Mod #179 (#228)
* Added ./statusd-data into .gitignore
* Increased log level for the test node from INFO to ERROR
* Add call to loop.Run to evaluate all setTimeout/setIntervals methods. (#208)
* Rebase onto geth1.6.7 (#232)
* Got back sync duration from 60s to 30s, updated bindata.go
2017-08-04 23:14:17 +07:00
Ewetumo Alexander b50c46caa8 Add SetTimeout/SetInterval to JailCell otto.Otto, closes #189
* Commit initial change for settimeout/setinterval

* Add initial tests for jail setTimeout/setInterval

* Add ottoext dependency

* Add fetch jail test with function

* Add dependencies of fetch from ottoext

* Refactor with regards to PR review

* Refactor with regards to PR review

* Fix syntax errors

* Fix missing return statement
2017-07-13 18:04:47 +07:00
Victor Farazdagi 6b3f7aabdf node: expose in-proc RPC (CallRPC), closes #144 2017-05-28 17:05:33 +03:00
Victor Farazdagi 349103de1a major refactoring: node manager, tx queue, accounts, jail
- node: signals and node reset, fixes #152
- tests update (testify is used)
- node manager refactored, race conditions fixed
- node wrapper has been removed, we rely on go-ethereum node now
- tx queue refactored
- jail refactored
- backend and API introduced
2017-05-24 00:29:20 +03:00
Victor Farazdagi 9c679d6bb7 verndor: stretchr/testify added 2017-05-23 06:04:03 +03:00