Commit Graph

536 Commits

Author SHA1 Message Date
Andrea Maria Piana b8ea79a3f0
Upgrade status-protocol-go (#1586) 2019-08-29 08:33:46 +02:00
dmitry 7454586889 Match transfer to every account that is being watched 2019-08-28 17:38:58 +03:00
Dmitry Shulyak 0165b028c9
Watch new accounts aftter they were saved to accounts table (#1569)
* Watch new accounts once they are saved in accounts table

* Add test that reactor can be restarted and watch new accounts
2019-08-28 10:49:03 +03:00
Andrea Maria Piana e118e89107
Add persistent datasync, use single database and handle negotiated filters (#1580) 2019-08-27 14:04:15 +02:00
Dmitry da5876f632 Add api method settings_getConfigs to retrieve list of configs in a batch quer 2019-08-22 17:32:44 +03:00
Dmitry a67184adbd Wallet database isolated by the network id
Wallet database refactored so that every query ensures isolation by the network id.
Network id provided when database object is created, thus it is transparent to other parts
of the wallet module.

Additionally every uniqueness index is changed to ensure that it doesn't prevent adding
object with same id but from a different network.
2019-08-21 10:44:50 +03:00
Dmitry 494cb5bb33 Create single database appdatase 2019-08-21 10:44:50 +03:00
Dmitry Shulyak be9c55bc16
Accounts data management (#1530)
* WIP accounts implementation

* Accounts datasore and changes to status mobile API

* Add library changes and method to update config

* Handle error after account selection

* Add two methods to start account to backend

* Use encrypted database for settings and add a service for them

* Resolve linter warning

* Bring back StartNode StopNode for tests

* Add sub accounts and get/save api

* Changes to accounts structure

* Login use root address and fetch necessary info from database

* Cover accounts store with tests

* Refactor in progress

* Initialize status keystore instance before starting ethereum node

* Rework library tests

* Resolve failures in private api test and send transaction test

* Pass pointer to initialized config to unmarshal

* Use multiaccounts/accounts naming consistently

Multiaccount is used as a login identifier
Account references an address and a key, if account is not watch-only.

* Add login timestamp stored in the database to accounts.Account object

* Add photo-path field for multiaccount struct

* Add multiaccoutns rpc with updateAccount method

Update to any other account that wasn't used for login will return an error

* Fix linter in services/accounts

* Select account before starting a node

* Save list of accounts on first login

* Pass account manager to accounts service to avoid selecting account before starting a node

* Add logs to login with save and regualr login
2019-08-20 18:38:40 +03:00
Andrea Maria Piana 06dc227071
Add Metadata to messages, expose new messenger methods (#1571)
* Add Metadata to messages, expose new messenger methods

This commits modifies deduplicator so that it takes a `StatusMessage`
instead of `WhisperMessage` and also returns a `Metadata` field which is
then passed back by the client when confirming messages, which fixes the
issue we had with not confirming pfs messages.
2019-08-20 13:20:25 +02:00
Adam Babik 8383feea04 add API to shhext to manage messages 2019-08-12 19:44:31 +02:00
Andrea Maria Piana 9ae7d2b6d5 Move envelopes tracking to status-protocol-go
This commit moves envelopes tracking to status-go.
Post endpoint is not going to track envelopes anymore, as that's taken
care on status-protocol-go side, so this is a breaking change, and
version is updated accordingly.
2019-08-12 19:44:31 +02:00
Andrea Maria Piana a3a413b5d9 Add SaveChat, Chats, DeleteChat, SaveConctat & Contacts endpoint 2019-08-05 16:03:18 +02:00
Adam Babik ce7b0fc376 conslidate protocol databases 2019-08-01 10:40:54 +02:00
Andrea Maria Piana 9de77b21b2 Add datasync, v1messages & disable discovery topic options
Adds support for datasync, V1Messages and disabling the discovery topic.
This is a backward compatible change as long as they are not toggled on
(they are not by default).
2019-07-29 19:39:44 +02:00
Dmitry Shulyak 6f1c2eca12
Don't omit history index when the value is 0 (#1545) 2019-07-27 08:48:23 +03:00
Andrea Franz acfe9a721c
Multi account login (#1525)
* multi-account login and signing

put methods count threshold back to 20

* validate login params

* refactoring

* use common.Address

* remove unused var in test
2019-07-26 16:45:10 +02:00
Adam Babik e93d994460 integrate status-protocol-go 2019-07-24 18:46:43 +02:00
Dmitry Shulyak 9723b64827
Permissions api (#1524)
* Add permissions api

* Integrate permissions service

* Reduce cyclomatic complexity of the MakeNode function
2019-07-21 08:41:30 +03:00
Dmitry Shulyak 9dbf5a0c86
Add browsers API (#1523)
Add browsers API
2019-07-17 08:28:37 +03:00
Dmitry Shulyak 40b6b3da13
Simplify transfer object (#1521)
* Store tx and receipt in db and cast it to TransferView on read

* Store Log instead of log index

* Use contract from log and bring back address field

* Add tx status and id fields
2019-07-15 14:16:07 +03:00
Dmitry Shulyak e20648ecc7
[services/wallet] Several changes in API after feedback (#1517)
[services/wallet] Several changes in API after feedback

- Timestamp from block header stored in blocks table and added to each transfers
- From field is computed from signature to offload this computation from client side
- `history` event is added back, so that client can ignore historical blocks when watching
only for new blocks
-  block number and timestamp are marshalled in hex. consistent with ethereum data structures
2019-07-10 12:08:43 +03:00
Andrea Maria Piana 03cf452b8f Fix initialization 2019-07-10 08:02:31 +02:00
Andrea Maria Piana fa639ffc74 Handle device not found
When receiving a message from someone not targeting our device,
we reply with an empty message that includes our own devices, so next
time they send a message they will include our device.
2019-07-08 12:03:14 +02:00
Andrea Maria Piana 2a2fcb48e1 Refactor publisher 2019-07-08 08:24:08 +02:00
Dmitry a2295baaf9 Add api method to query token balances 2019-07-05 08:38:29 +03:00
Andrea Maria Piana f2139105bf Add personal filter
Add a filter for 1-to-1, currently not used but useful in potential
migrations
2019-07-04 14:49:44 +02:00
Andrea Maria Piana b3efbb54f8 Send signal on new messages 2019-07-04 14:49:44 +02:00
Adam Babik dbaf622e12
Refactor messaging (#1510)
This change flattens messaging/chat package. It also removes dependency between multidevice and chat/protobuf packages.

The Publisher interface was also changed a bit to support more native types.

Version got bumped to 0.29.0-beta.3.
2019-07-03 21:13:11 +02:00
Adam Babik aa39647094
Publisher refactor (#1508) 2019-07-01 11:39:51 +02:00
Adam Babik e28d4ef1a3
Create messaging package and move some packages into it (#1505) 2019-06-26 20:17:41 +02:00
Adam Babik 1ab2e88bf5
Sync messages in a loop until error or no cursor (#1502) 2019-06-26 18:17:41 +02:00
Andrea Maria Piana 5335a2b4fd
Move installations to status-go (#1499)
* Move installations to status-go

This commit moves installations management/storage to status-go.
We remove the native binding and provide RPC endpoints to set the
metadata and return a list of our own installations.
2019-06-26 11:32:59 +02:00
Andrea Maria Piana 0ade9a6cbb
Cache keys (#1497)
* Cache keys

Generating a symkey can take up to a second on slow devices, this commit
makes so that keys are saved once generated and stored in the database.
2019-06-24 09:26:25 +02:00
Andrea Maria Piana f6fba1d3d6 Publish contact code periodically 2019-06-19 17:38:45 +02:00
Andrea Maria Piana 1aa3e2812a Add support for partitioned topic 2019-06-19 17:38:45 +02:00
Andrea Maria Piana 78ed35d2fe Add protocol version to bundle 2019-06-19 17:38:45 +02:00
Andrea Maria Piana cef7f367ab Add topic negotiation
This commit add topic negotiation to the protocol.

On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.

This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).

When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.

This change is backward compatible, as long as there is no downgrade of
the app on the other side.

A few changes:

* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).

* Factored out multidevice management in a separate namespace

* Moved all the test to test the whole protoocl rather than just the encryption service

* Moved all the filter management in status-go
2019-06-19 17:38:45 +02:00
Dmitry Shulyak 047c9b5263
Download transfers starting from latest block header (#1467) 2019-06-14 13:16:30 +03:00
Adam Babik 1c33220834 update linter 2019-06-12 13:12:00 +02:00
Adam Babik c9e99c432d migrate to go 1.12 and go modules 2019-06-12 13:12:00 +02:00
Dmitry Shulyak 4cafe63ab7
Unmarshall wrapped into string integers as string (#1470) 2019-05-24 13:22:13 +03:00
Dmitry Shulyak d32cd18c09
Fix channel subscriber that was blocking sending new requests (#1465)
In RequestMessagesSync subscriber is listening to a feed where all whisper
events are posted. After we received event with a request hash - subscriber will
stop actively consuming messages from a feed, as a subscription channel will
get overflow and whole feed will get blocked.

Some events are posted to a feed before request is sent, so blocked feed results
in blocked sending.

Now we will unsubscribe after relevant event was received, and terminate subscriber
explicitly by timeout.
2019-05-20 11:10:26 +03:00
Andrea Maria Piana 4ab08629f6 Add postgres
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
2019-05-15 11:01:34 +02:00
Igor Mandrigin 10fc860a5f
fix parameters parsing for eth_* filters subscriptions (#1461) 2019-05-10 12:56:58 +02:00
Andrea Maria Piana 061f10e58d
Publish rlp.RawValue instead of envelope (#1459)
As part of a performance profiling of mailserver we noticed that most of
the resources on a query are spend decoding the whisper envelope.
This PR changes the way we store envelopes encoding the Topic into the
database key, so we can check that and we are able to publish the
envelope rawValue if it matches.
The change is backward compatible as only newly added envelopes will
have the new key, while old ones will have to be unmarshaled.
2019-05-09 12:58:02 +02:00
Igor Mandrigin 4c1f96d255
Implement subscription for eth and shh filters using signals. (#1455) 2019-05-07 09:05:38 +02:00
Dmitry Shulyak 14c513bd5a
Execute writes atomically only after request was processed without errors (#1454)
* Replace request ID when same request is restarted

* Remove unnecessary changes

* Execute all writes atomically only if request was processed succesfully

* Fix linter

* Fix shadowed errors

* Fix spelling

* Do not append same reference to a byte slice
2019-05-06 09:33:19 +03:00
Dmitry Shulyak cba00153e2 Replace request ID when same request is restarted (#1453) 2019-05-01 12:33:18 +02:00
Dmitry Shulyak 218a35e609
Notify users that envelope was discarded and retry sending it (#1446)
API for querying mail servers that skips already received data on subsequent requests
2019-04-30 09:46:12 +03:00
Andrea Maria Piana 354e6981ba
Add network incentivisation service (#1452) 2019-04-29 14:05:49 +02:00
Dmitry Shulyak a904d9325e
Notify users that envelope was discarded and retry sending it (#1424)
* Notify users that envelope was discarded and retry sending it

* Update Gopkg files with released whisper version

* Forgot to remove signal after refactoring
2019-04-02 13:40:45 +03:00
Andrea Franz 8fe14e8f23
hash typed data (#1426)
* add HashTypedData to backend

* add HashTypedData to backand, lib, and mobile

* rename typeddata.Hash to typeddata.ValidateAndHash
2019-03-28 15:56:21 +01:00
Adam Babik 4b790adf34
add shhext_requestMessagesNew (#1412) 2019-03-15 14:27:08 +01:00
Dmitry Shulyak aa28f652e3 Re-send user payload if previously sent envelope wasn't acknowledged (#1386)
* Split shhext.tracker into envelopes and mail monitors

* Send envelopes on every new attempt to deliver a message

* Re-send user payloads if previous envelopes weren't acknowledged

* Remove debug api across the codebase
2019-03-01 14:36:21 +01:00
Andrea Maria Piana 38e5335e18
Allow messages to be decrypted from previous bundles (#1400)
Currently we only decrypt messages if received on the current bundle.
This changes the behavior so that messages can be decrypted if sent to
previous bundles as well, as otherwise is a bit restrictive
2019-02-28 13:09:43 +01:00
Dmitry ccce8b9a80 Clear all request internal cache when node is stopped 2019-02-27 07:44:38 +02:00
Andrea Maria Piana e88e590028
Fix wrong datadir in desktop (#1389)
Desktop was wrongly passing a network dependent data-dir (mainnet_rpc).
The code migrates the db if present, moving it to the new location.
2019-02-25 08:51:25 +01:00
Dmitry f2a400dc44 Split shhext.tracker into envelopes and mail monitors 2019-02-22 09:33:46 +02:00
Dmitry Shulyak 224bc7857c Mark envelope as confirmed when it was received from a mail server (#1378) 2019-02-20 18:10:59 +01:00
Dmitry 381a1941a0 Unregister request whisper failed to send it 2019-02-20 16:09:45 +02:00
Andrea Maria Piana 81d8ca82a2
Add confirm messages processed by ID (#1375)
Currently PFS messages are decrypted and therefore modified before being
passed to the client. This make IDs computation difficult, as we pass
the whole object to the client and expect the object be passed back once
confirmed.
This changes the behavior allowing confirmation by ID, which is passed
to the client instead of the raw object.
This is a breaking change, but status-react is already forward
compatible.
2019-02-19 13:58:42 +01:00
Andrea Maria Piana 2a4382369a
Add GetContactCode call, add DH flag (#1367)
This PR does a few things:

1) Add a call GetContactCode to check whether we have a bundle for a
given user.

2) Add a DH flag to the API (non-breaking change), for those messages
that we want to target all devices (contact-requests for example).

3) Fixes a few small issues with installations, namely if for example a
messages is sent without a bundle (currently not done by any client),
we still infer installation info, so that we can communicate securely
and making it truly optional.
2019-02-12 12:07:13 +01:00
Ivan Daniluk 4f3f5ee574 Gomobile support (#1164) 2019-02-01 18:02:52 +01:00
Andrea Franz 0e0c3cd859
add shhext.InitProtocolWithEncryptionKey (#1362)
* add shhext.InitProtocolWithEncryptionKey

* update test to avoid lint error
2019-01-25 11:31:51 +01:00
Dmitry c8a616688c Add request with retries api call 2019-01-24 14:31:56 +02:00
Andrea Maria Piana 2df5422337
Add backward compatibility for single bundle (#1355)
We change the protocol to accomodate publishing multiple bundles, in
order to propagate bundles for group chats and have a way to extend it
further.
This commit re-introduces backward compatibility for direct messages,
to be removed once that is not an issue anymore.
2019-01-21 12:33:18 +01:00
Andrea Maria Piana ad51c019e8
Fix broken migration (#1354)
The migration from unencrypted->encrypted did not work properly, it was
deleting and re-creating the database. This correctly migrates the db.
2019-01-19 08:30:05 +01:00
Dmitry aa09d854ca Prevent frequent requests with same topics 2019-01-18 14:52:33 +02:00
Andrea Franz 4939268edf
Unlock wallet and chat keys (#1346)
* select account decrypting wallet and chat keys

* adapt account tests to use chat and wallet account/keys

* fix tests using chat address

* changes after review

* fix status service e2e tests

* add account.Info struct returned when creating and recovering an account

* use s.EqualValues to compare recovered accounts

* return Info instead of *Info

* add both address and walletAddress to responses

* Update lib/types.go

Co-Authored-By: gravityblast <andrea@gravityblast.com>

* Update lib/types.go

Co-Authored-By: gravityblast <andrea@gravityblast.com>

* update comment to fix lint
2019-01-18 10:01:14 +01:00
Dmitry Shulyak 863d07f798 Accept status service config directly from json input (#1337) 2019-01-17 13:56:22 +01:00
Adam Babik fd48ad8b29
geth: upgrade geth to 1.8.21 (#1350) 2019-01-16 15:42:00 +01:00
Andrea Maria Piana 59426d628c
Migrate from nokey on ios to re-keyed (#1345) 2019-01-11 11:55:18 +01:00
Roman Volosovskyi :: Darkviolet Lightgreen Halcyon d6216f1aa4 [slow sign in] Reduce number of kdf iterations for PDF database key (#1343)
`kdf_iter` parameter is reduced to 3200. This change is done because of
performance reasons, currently key derivation is too slow on some mobile
devices. The number of iterations before this commit is 64000, default
value in `sqlcipher` from version `3.0.0`.
fda4c68bb4/CHANGELOG.md (300---2013-11-05)

Implementation:
`sqlcipher_export` is used for migration, check out the link below
for details
https://www.zetetic.net/sqlcipher/sqlcipher-api/#sqlcipher_export
2019-01-11 11:12:23 +01:00
Adam Babik fa97e6927d
Add CreateAccount RPC function (#1330) (#1334) 2018-12-28 12:37:22 +01:00
Andrea Maria Piana 30dbacdcca
Initialize public key (#1332) 2018-12-24 08:18:27 +01:00
Andrea Maria Piana 3292538b41
Allow multiple bundles, change n devices & bundle refresh (#1331)
Change to support sending multiple bundles, as needed for group chats,
limit number of devices to 3 as already done in the UI and refresh
bundle daily.
2018-12-21 11:07:25 +01:00
Andrea Maria Piana 79bf92bea5
Allow specify topic in rpc endpoint (#1328) 2018-12-18 16:27:12 +01:00
Adam Babik 74cb16c456
add SyncMessages method to shhext api (#1309) 2018-12-14 12:21:34 +01:00
Dmitry Shulyak f2c6fef64c
Persist selected mail server using separate monitor (#1303)
This change allows to connect to the mail server that we were using before the app was restarted. Separate loop is listening for whisper events, and when we receive event that request was completed we will update time on a peer record.

Records are stored in leveldb. Body of the record is marshaled using json. At this point the only field is a timestamp when record was used.

This loop doesn't control connections, it only tracks what mail server we ended up using. It works asynchronously to connection management loop. Which tracks events that are related to connection state and expiry of the requests.

When app starts we look into the database and select the most recently used record. This record is added to connection management loop first. So if this server is available we will stick to using it. If we weren't able to connect to the same server in configured timeout (5s) we will try to connect to any other server from list of active servers.

closes: #1285
2018-12-12 11:39:00 +02:00
Adam Babik e2682486fd
Fix cursor encoding (#1308)
It fixes encoding and decoding cursor sent in historic messages requests. `hex` package from the standard library is used.
2018-12-11 11:23:47 +01:00
Andrea Maria Piana b676de9dac
Hash password before using it (#1306) 2018-12-10 14:11:19 +01:00
Adam Babik db691fd763
fix golang/mock (#1310) 2018-12-10 13:53:27 +01:00
Dmitry Shulyak d51761f83e Ensure sent event is received from a mailserver (#1304)
In previous change i used same filtering for a handler that processes confirmations.
But if we are connected with peers that do not support confirmations on whisper level,
whisper may send "Sent" event when envelope is written to a socket. This is our old behaviour.

So, this PR allows to use confirmations from a mail servers and ensure that transition between
multiple version of peers will be smooth.
2018-12-06 10:30:57 +01:00
Dmitry Shulyak a609b468fe
Mail peer store and connection manager (#1295)
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history

When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.

If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.

If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.

We will have two components that will rely on this logic:
1. requesting history

If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.

2. confirmation from selected mail server

Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.

I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server

Currently this feature is hidden, as certain changes will be necessary in status-react. 

partially implements: https://github.com/status-im/status-go/issues/1285
2018-12-05 15:57:05 +02:00
Andrea Maria Piana afc3017e07
Notify user when the device is missing (#1298)
* Notify user when the device is missing

* Update services/shhext/chat/encryption.go

Co-Authored-By: cammellos <andrea.maria.piana@gmail.com>
2018-12-05 09:22:49 +01:00
Andrea Maria Piana 38bb4d8ef3
Add versioning & tests, migrate db files (#1293)
We are preparing for the release of this to general public, so a few
things have been added:

1) Add versioning for bundles, and make refresh interval configurable
2) Move files to installationID so no metadata is leaked
3) Re-key using user password db
2018-11-28 12:34:39 +01:00
Dmitry e60dbe3c1b Update selected mail servers that are used in envelope tracker 2018-11-28 11:16:37 +02:00
Andrea Maria Piana 6112ca0289
Add tests for multi-device and refactor encryption service config (#1277) 2018-11-27 09:54:20 +01:00
Dmitry Shulyak aac706fe4c
Use confirmation that bundle is delivered for sending feedback to user (#1284)
* Pull whisper confirmations changes

* Use batch confirmations as a signal that envelope was sent into the network
2018-11-27 08:30:15 +02:00
Adam Babik 52a1bdfed6
Upgrade geth 1.8.17 plus add metrics during compilation time (#1273)
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.

The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
2018-11-14 08:03:58 +01:00
Andrea Maria Piana 1f6cccd0fc
Add enabling/disabling of installations (#1264)
This commit adds a list new table, installations, which is used to keep
track of which installation are active for a given identity key.

In general, we limit the number of installation that we keep
synchronized to 5, to avoid excessive usage of resources.

Any installation coming from our own identity, will have to be manually
enabled, otherwise we trust the other peer has correctly paired their
devices.

We use a timestamp to decide which installations to keep synchronized as
a logical clock would have make the creation of the bundle more
complicated, but this can always be converted to a logical clock at
later stages without breaking compatibility.
2018-11-06 09:05:32 +01:00
Dmitry Shulyak 3fe5b25ff2
Implement eth_getFilterLogs (#1265)
Queries logs from remote server using original filter criteria.
2018-11-06 07:41:36 +01:00
Dmitry Shulyak db786ef1d2
Sign typed data implementation (#1250)
* Implement EIP 712

* Cover majority of cases with tests

* Add solidity and signer tests and cover integer edges case

* Add thin api to sign type data using selected status account

* All integers are extended to int256 and marshalled into big.Int

* Document how deps works

* Fix linter

* Fix errors test

* Add validation tests

* Unmarshal every atomic type in separate functions
2018-11-06 07:26:12 +01:00
Andrea Maria Piana ee3c05c79b
Change handling of skipped/deleted keys & add version (#1261)
- Skipped keys

The purpose of limiting the number of skipped keys generated is to avoid a dos
attack whereby an attacker would send a large N, forcing the device to
compute all the keys between currentN..N .

Previously the logic for handling skipped keys was:

- If in the current receiving chain there are more than maxSkip keys,
throw an error

This is problematic as in long-lived session dropped/unreceived messages starts
piling up, eventually reaching the threshold (1000 dropped/unreceived
messages).

This logic has been changed to be more inline with signals spec, and now
it is:

- If N is > currentN + maxSkip, throw an error

The purpose of limiting the number of skipped keys stored is to avoid a dos
attack whereby an attacker would force us to store a large number of
keys, filling up our storage.

Previously the logic for handling old keys was:

- Once you have maxKeep ratchet steps, delete any key from
currentRatchet - maxKeep.

This, in combination with the maxSkip implementation, capped the number of stored keys to
maxSkip * maxKeep.

The logic has been changed to:

- Keep a maximum of MaxMessageKeysPerSession

and additionally we delete any key that has a sequence number <
currentSeqNum - maxKeep

- Version

We check now the version of the bundle so that when we get a bundle from
the same installationID with a higher version, we mark the previous
bundle as expired and use the new bundle the next time a message is sent
2018-11-05 20:00:04 +01:00
Dmitry Shulyak 539fa01d48
Implement cache for recording removed logs due to reorg (#1241)
Adjust criteria if replaced logs were received and add more tests

Improve validation

Changes after review
2018-10-28 17:12:16 +01:00
Andrea Maria Piana 22eb53ea8c shhext: increase max skip keys (#1257) 2018-10-26 10:44:10 +02:00
b00ris bdbceba6eb
added ulc config params (#1255)
* added ulc config params

* fmt

* fix lint
2018-10-24 19:31:41 +03:00
Dmitry Shulyak 29b55bd445
Add local implementation of the newFilter call using remote getLogs call (#1235)
* Implement subscriptions and filtering

* Add e2e test with log filter polling logs from EVM with clique backend

* Apply review comments

* Move devnode to t/devtests to avoid cycle in imports
2018-10-23 08:11:11 +03:00
Adam Babik a75f9c34cf
Request historical messages in batches (#1245) 2018-10-19 11:09:13 +02:00
Igor Mandrigin ba504e99c4
Mailserver: return error response. (#1244) 2018-10-18 12:25:00 +02:00
Ivan Daniluk e7c588fcd0
Change API of BuildDirectMessage function to use variadic arguments (#1243) 2018-10-16 17:22:28 +02:00
Andrea Maria Piana e4ba365b8a
Add bundles.added signal & pairing endpoint (#1237) 2018-10-16 12:31:05 +02:00
Andrea Franz c86f8bf6ca
allow multiple topics in mailserver requests (#1240)
* allow multiple topics in mailserver requests

* simplify topicsToBloom function

* create test topics simply from the string without hash

* add deprecated comment to MessagesRequest.Topic
2018-10-15 23:15:04 +02:00
Andrea Maria Piana 9f8f0089a3
Add extract group membership signatures (#1226) 2018-10-04 18:53:48 +02:00
Dmitry 5b551c67fa Use status-im whisper/whisperv6 module everywhere 2018-10-03 09:27:51 +03:00
Andrea Maria Piana b309718fdc
Add sign & verify api calls (#1218) 2018-09-27 15:07:32 +02:00
Andrea Maria Piana dcaf8caed0
Add x3dh key exchange (#1127)
* Add x3dh key exchange

* Encrypt using the double ratchet

* Multi device with auto-pairing

* Add pfs enabled flag
2018-09-24 20:07:34 +02:00
Sebastian Delgado 4afd9e6c6c Remove transactions queue 1027 (#1125)
Remove `PendingSignRequests` queue from the sign module.

This closes #1027 by removing the pending sign requests queue dependency from the SendTransaction, SignMessage and Recover.
2018-08-16 13:37:53 +02:00
Pedro Pombeiro 840b5b64d2 Implement mailserver canary service. Closes #1086 2018-08-07 17:17:59 +02:00
Sebastian Delgado 7577296b3c
Implement eth_newPendingTransactionFilter (#1113)
Implement eth_newPendingTransactionFilter
2018-07-27 08:54:40 -07:00
Adrià Cidre bfbb02019f
Discover mail servers on demand. (#1082)
* [#1076] Discover mail servers on start up.

* [#1076] On-demand mail-server discovery
2018-07-16 09:40:40 +02:00
Adrià Cidre d61c39bbd8
Fix timeout issues with TestRequestMessagesSuccess (#1091) 2018-07-11 16:42:51 +02:00
Adam Babik 38a60135b2 Add asymmetric key support for MailServer requests (#1075)
* add Asymmetric Key support for MailServer requests

* remove deprecated notice

* fix linter

* refactoring Whisper config related to MailServer

* fix race condition
2018-07-04 11:30:57 +02:00
Andrea Franz 809db97e54
mailserver pagination (#1039)
* mailserver sends envelopes in descending order

* add limit value in mailserver request payload

* mailserver sends messages up to the limit specified in the request

* update Archive method to return key and error

* processRequest returns the next page cursor

* add cursor to mailserver request

* add limit and cursor to request payload

* fix request limit encoding

* wait for request completed event in TrackerSuite/TestRequestCompleted

* add cursor to mailserver response

* fix cursor position in payload

* add e2e test for mail server pagination

* validate mail server response size

* remove old limitReached var

* fix lint warnings

* add whisper patch

* fix tests after rebase

* check all return values to avoid lint warnings

* check that all messages have been retrieved after 2 paginated requests

* fix lint warnings

* rename geth patch

* merge mailserver patches into one

* add last envelope hash to mailserver response and EventEnvelopeAvailable event

* update whisper patch

* add docs to MailServerResponse

* update whisper patch

* fix tests and lint warnings

* send mailserver response data on EventMailServerRequestCompleted signal

* update tracker tests

* optimise pagination test waiting for mailserver to archive only before requesting

* rollback mailserver interface changes

* refactoring and docs changes

* fix payload size check to determine if a limit is specified

* add more docs to the processRequest method

* add constants for request payload field lengths

* add const noLimits to specify that limit=0 means no limits
2018-07-02 09:38:10 +02:00
Andrea Franz 1bc63df5f2
validate query range on requestMessages API (#1060) 2018-06-26 10:41:03 +02:00
Adrià Cidre 5bbfabde94
New debug_postSync endpoint (#1046)
* [#ISSUE-1041] New debug_postconfirm endpoint

* move debug service api inside shhext service
2018-06-25 15:27:17 +02:00
Andrea Franz fa390a52ae
RequestHistoricMessages response (#1009)
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Pedro Pombeiro 354e23aaf5 Fix issues reported by lint. Part of #1017 2018-06-14 13:52:51 +02:00
Adrià Cidre d5be8c525d
[#856] move geth subpackages to root level (#1007) 2018-06-08 13:29:50 +02:00
Adrià Cidre c82d5635b6
Allow gas and gas input as argument on signing TTXX (#1000)
* Allow gas and gas input as argument on signing TTXX

* Two different bindings ApproveSignRequest and ApproveSignRequestWithArgs
2018-06-06 09:35:32 +02:00
Adam Babik 717d0fbadb
Fix another round of flaky tests (#996) 2018-05-29 13:24:23 +02:00
Andrea Franz 6c469299b6
fix default range value for shhext_requestMessages (#934)
* fix default range value for shhext_requestMessages

* use t.Run to run scenarions

* fix case when To - 24 hours is negative
2018-05-10 16:47:54 +02:00
Igor Mandrigin a933885806 Deduplicator: add API to confirm received messages. 2018-05-10 15:28:48 +02:00
Igor Mandrigin 0241f95d6a Support chain diverges in block filters. 2018-05-09 18:07:26 +02:00
Igor Mandrigin 31cf2297d2 Implement block filters API for the RPC mode.
Some operations (like deploying contracts) require filter APIs to work.
Since these operations aren't supported on Infura anymore, and we don't
run LES, a separate implemenation of filters is required.

Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
2018-05-08 19:46:25 +02:00
Adrià Cidre 246c5321ac
Fix services mock generation (#901) 2018-05-07 10:22:38 +02:00
Pedro Pombeiro 5d802abe9e Increase timeouts in order to accommodate `-race` flag
- Fixes https://jenkins.status.im/job/status-go/job/race-check/56/console
2018-05-04 14:02:13 +02:00
Dmitry Shulyak 707221954f
Make whisper tolerant to local time skews (#864)
This change adds adds an ability to use different source of time for whisper:

when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.

So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.

In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.

Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.

Closes: #687
2018-05-04 11:23:38 +03:00
Adrià Cidre 8c9db81bec
Expose status specific methods (login/signup/joinpublicchannel) through the RPC api (#877) 2018-05-03 12:36:56 +02:00
Ivan Danyliuk 953c26e8cf
Move signal logic into signal package 2018-05-03 09:35:58 +02:00
Igor Mandrigin f4cd8d27b5 Add `shhext_getNewFilterMessages` function to RPC APIs.
This function returns only the new messages from the filter, never
returns the same message for the same user twice.
2018-04-27 15:24:37 +02:00
Frank Mueller 92e02189c7
Add doc for shhext_requestMessages (#872) 2018-04-27 13:59:01 +02:00
Frank Mueller b543d32a31
Move ssh_requestMessages to sshext_requestMessages (#862)
* Add RequestMessage to sshext

* E2E tests now use shhext_requestMessages

* Typo in comment

* Enhanced maintainability

* Drop former mailservice

* Code reorg after review

* Fix missed changes after update to 1.8.5
2018-04-26 07:56:19 +02:00
Dmitry Shulyak b37fda7731
Update go-ethereum to 1.8.5 (#854)
* Rebase on 1.8.5

* Remove outdated patches and apply all others

* Use shh_post that returns hash

* Use bloom filter for request to mailserver

* Remove tests for sending messages without subbing first

* Fix deadlock in ethdb

* Expect null if receipt is not yet created

* Subscribe to messages before sending them in whisper test
2018-04-24 18:50:26 +03:00
Igor Mandrigin f0f55d408f Make sure that only `personal_sign` and `personal_ecRecover` are exported. 2018-04-18 18:22:21 +02:00
Igor Mandrigin 4cc6028d59 Implement `personal_sign`.
This commit implements `personal_sign` RPC or web3 personal.sign
methods.

NB! Contains breaking API changes.
2018-04-17 11:18:58 +02:00