Commit Graph

310 Commits

Author SHA1 Message Date
Andrea Maria Piana 141f917e7c
Initialize protocol when waku active () 2020-02-17 15:38:59 +01:00
Volodymyr Kozieiev 19fc448c85
pending-stickers field added to settings, unit-tests fixed () 2020-02-13 13:48:04 +02:00
Roman Volosovskyi 9cf640842b
[] Upgradable paths in configs
Storing absolute path for different configs breaks compatibility on iOS
as app's dir is changed after upgrade. The solution is to store relative
paths and to concatenate it with `backend.rootDataDir`. The only
exception is `LogFile` as it is stored outside `backend.rootDataDir` on
Android. `LogDir` config was added to allow adding of custom dir for log
file.
Configs concerned:
`DataDir`
`LogDir`
`LogFile`
`KeystoreDir`
`BackupDisabledDataDir`
2020-02-03 20:02:01 +02:00
Roman Volosovskyi a92a95cf83
Faster tx fetching with less request
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
  each address of a new empty multiaccount this scan required
  1. two `eth_getBalance` requests to find out that there is no any
     balance change between zero and the last block, for eth transfers
  2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
     for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
  also accounts were not deduplicated. So even for an empty multiacc we
  scanned chain twice for each chat and main wallet addresses, in result
  app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
  there were any eth transfers, and that caused empty history in case
  if user already used all available eth (so that both zero and latest
  blocks show 0 eth for an address). There might have been transactions
  but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
  scan, so it could require indefinite number of requests; the scanning
  algorithm was written so that we persisted the whole history of
  transactions or tried to scan form the beginning again in case of
  failure, giving up only after 10 minutes of failures. In result
  addresses with sufficient number of transactions would never be fully
  scanned and during these 10 minutes app could use gigabytes of
  internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
  requests. These requests return significantly bigger responses than
  `eth_getBalance`/`eth_transactionsCount` and it is likely that
  execution of thousands of them in parallel caused failures for
  accounts with hundreds of transactions. Even for an account with 12k
  we could successfully determine blocks with transaction in a few
  minutes using `eth_getBalance` requests, but `eth_getBlock...`
  couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
  caused in average 3-4 times more such requests than is needed.

*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
  then check erc20 in the range where these eth transactions were found.
  For an empty address in multiacc this means:
  1. two `eth_getBalance` transactions to determine that there was no
     balance change between zero and the last block; two
     `eth_transactionsCount` requests to determine there are no outgoing
     transactions for this address; total 4 requests for eth transfers
  2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
     but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
  account, so a new multiacc requires ~25 (we also request latest block
  number and probably execute a few other calls) request to determine
  that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
  1. determine the range which contains 20-25 outgoing eth/erc20
     transactions. This usually requires up to 10 `eth_transactionCount`
     requests
  2. then we scan chain for eth transfers using `eth_getBalance` and
     `eth_transactionCount` (for double checking zero balances)
  3. we make sure that we do not scan db for more than 30 blocks with
     transfers. That's important for accounts with mostly incoming
     transactions, because the range found on the first step might
     contain any number of incoming transfers, but only 20-25 outgoing
     transactions
  4. when we found ~30 blocks in a given range, we update initial
     range `from` block using the oldest found block
  5. and now we scan db for erc20transfers using `eth_getLogs`
     `oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
  6. when all blocks which contain incoming/outgoing transfers for a
     given address are found, we save these blocks to db and mark that
     transfers from these blocks are still to be fetched
  7. Then we select latest ~30 (the number can be adjusted) blocks from
     these which were found and fetch transfers, this requires 3-4
     requests per transfer.
  8. we persist scanned range so that we know were to start next time
  9. we dispatch an event which tells client that transactions are found
  10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
  20 transfers, if not we scan chain again and return transfers after

small fixes
2020-01-23 10:36:11 +02:00
Adam Babik bc2d018483
Add support for request messages by topics () 2020-01-21 08:11:24 +01:00
Adam Babik 79b8112f89
Split shhext into shhext and wakuext () 2020-01-20 21:56:06 +01:00
Adam Babik 44aa313981
Make shhext and protocol work with Waku ()
This change makes shhext and protocol submodule work with Waku and Whisper.
2020-01-13 20:17:30 +01:00
Andrea Maria Piana 88a1d0111e
Add Commands ()
This commit adds handling of Transaction commands.
2020-01-10 19:59:01 +01:00
Adam Babik 37a508a97b
Integrate Waku service () 2020-01-08 12:12:23 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 287e5cdf79 Abstract `accounts.Key` and geth `keystore` 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 18af9175ac Prevent crash in TestWhisperTestSuite when run isolated 2019-12-30 12:56:05 +01:00
Adam Babik 1ac515f19e remove reSelectAccount from backend, SelectKeyPair and SelectedKeyPairID methods from Whisper 2019-12-30 12:56:05 +01:00
yenda 024f30f0b9 Put settings in a table with schema ()
Move settings table schema from a key-value store to a one row table with many columns.

We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.

Co-authored-by: Adam Babik <a.babik@designfortress.com>
2019-12-27 10:58:25 +01:00
Pedro Pombeiro dd894ece15 Start abstracting geth Keystore 2019-12-19 14:11:48 +01:00
André Medeiros 0571f561f0
APIs to Get and Create custom Tokens () 2019-12-10 12:31:08 -05:00
Adam Babik a636f33109
Set Whisper version submodule to v6 () 2019-12-09 11:36:14 +01:00
Adam Babik 4ac4a61e20
Move whisper to status-go monorepo () 2019-12-09 11:06:04 +01:00
Roman Volosovskyi 4c0d8dedea
Replace address with keyUid in accounts db
Account's address was used as a primary key in accounts db and as a
deterministic id of an account in some API calls. Also it was used as a
part of the name of the account specific database. This revealed some
extra information about the account and wasn't necessary.
At first the hash of the address was planned to be used as a
deterministic id, but we already have a keyUid which is calculated as
sha256 hash of account's public key and has similar properties:
- it is deterministic
- doesn't reveal accounts public key or address in plain
2019-12-09 11:20:12 +02:00
Andrea Maria Piana fd49b0140e
Move to protobuf for Message type ()
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`

* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.

* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.

* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
2019-12-05 17:25:34 +01:00
Pedro Pombeiro 26b6d7c36a Create a home submodule for Eth node bridges- Rename StatusBackend to GethStatusBackend 2019-11-27 17:02:09 +01:00
Pedro Pombeiro ed5a5c154d
Move to monorepo structure ()
Move to a monorepo structure with submodules

- Rename status-protocol-go to status-go/protocol
2019-11-21 17:19:22 +01:00
Pedro Pombeiro 2dd74da23d Integrate Nimbus status-protocol-go 2019-11-18 14:30:03 +01:00
Pedro Pombeiro c874960215
Use status-protocol-go/bridge/geth ()
* Use status-protocol-go/bridge/geth
2019-10-14 09:53:38 +02:00
Pedro Pombeiro a2f106e4c5
Update for status-protocol-go API changes () 2019-10-09 16:22:53 +02:00
Adam Babik 26880b83d7
Upgrade geth to 1.9.5 and Whisper () 2019-10-04 17:21:24 +02:00
Andrea Maria Piana b27779aa4e
Add Mailserver Request Gaps service ()
This commits add some endpoints to store mailserver request gaps,
and provides 4 endpoints:

```
AddMailserverRequestGaps(gaps []MailserverRequestGap) error
GetMailserverRequestGaps(chatID string) []MailserverRequestGap
DeleteMailserverRequestGaps(ids []string) error
DeleteMailserverRequestGapsByChatID(id string) error
```
2019-09-04 12:04:17 +02:00
Adam Babik d7f454fe4c
Add mailservers service () 2019-09-04 08:25:33 +02:00
dmitry bf95a71f6f Verify valid sub-accounts using database 2019-08-30 14:11:00 +03:00
Dmitry 494cb5bb33 Create single database appdatase 2019-08-21 10:44:50 +03:00
Dmitry Shulyak be9c55bc16
Accounts data management ()
* WIP accounts implementation

* Accounts datasore and changes to status mobile API

* Add library changes and method to update config

* Handle error after account selection

* Add two methods to start account to backend

* Use encrypted database for settings and add a service for them

* Resolve linter warning

* Bring back StartNode StopNode for tests

* Add sub accounts and get/save api

* Changes to accounts structure

* Login use root address and fetch necessary info from database

* Cover accounts store with tests

* Refactor in progress

* Initialize status keystore instance before starting ethereum node

* Rework library tests

* Resolve failures in private api test and send transaction test

* Pass pointer to initialized config to unmarshal

* Use multiaccounts/accounts naming consistently

Multiaccount is used as a login identifier
Account references an address and a key, if account is not watch-only.

* Add login timestamp stored in the database to accounts.Account object

* Add photo-path field for multiaccount struct

* Add multiaccoutns rpc with updateAccount method

Update to any other account that wasn't used for login will return an error

* Fix linter in services/accounts

* Select account before starting a node

* Save list of accounts on first login

* Pass account manager to accounts service to avoid selecting account before starting a node

* Add logs to login with save and regualr login
2019-08-20 18:38:40 +03:00
Andrea Maria Piana 9ae7d2b6d5 Move envelopes tracking to status-protocol-go
This commit moves envelopes tracking to status-go.
Post endpoint is not going to track envelopes anymore, as that's taken
care on status-protocol-go side, so this is a breaking change, and
version is updated accordingly.
2019-08-12 19:44:31 +02:00
Andrea Maria Piana cb769ccca9 Revert "Initialize status keystore instance before starting ethereum node"
This reverts commit 476240fbd1.
2019-08-05 16:03:18 +02:00
Dmitry 476240fbd1 Initialize status keystore instance before starting ethereum node 2019-08-01 11:01:02 +03:00
Andrea Franz acfe9a721c
Multi account login ()
* multi-account login and signing

put methods count threshold back to 20

* validate login params

* refactoring

* use common.Address

* remove unused var in test
2019-07-26 16:45:10 +02:00
Andrea Franz dcb0fa5262
add multi-account support ()
* add multiaccount support

add multi account ImportPrivateKey and StoreAccount

test derivation from normal keys

* add multiaccount to mobile pkg

* use multiaccount params structs from the mobile pkg

* move multiaccount tests together with the other lib tests

* fix codeclimate warning and temporarily increase methods threshold

* split library_test_utils.go to avoid linter warnings
2019-07-24 20:59:15 +02:00
Dmitry Shulyak 40b6b3da13
Simplify transfer object ()
* Store tx and receipt in db and cast it to TransferView on read

* Store Log instead of log index

* Use contract from log and bring back address field

* Add tx status and id fields
2019-07-15 14:16:07 +03:00
Dmitry Shulyak e20648ecc7
[services/wallet] Several changes in API after feedback ()
[services/wallet] Several changes in API after feedback

- Timestamp from block header stored in blocks table and added to each transfers
- From field is computed from signature to offload this computation from client side
- `history` event is added back, so that client can ignore historical blocks when watching
only for new blocks
-  block number and timestamp are marshalled in hex. consistent with ethereum data structures
2019-07-10 12:08:43 +03:00
Andrea Maria Piana cef7f367ab Add topic negotiation
This commit add topic negotiation to the protocol.

On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.

This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).

When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.

This change is backward compatible, as long as there is no downgrade of
the app on the other side.

A few changes:

* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).

* Factored out multidevice management in a separate namespace

* Moved all the test to test the whole protoocl rather than just the encryption service

* Moved all the filter management in status-go
2019-06-19 17:38:45 +02:00
Dmitry Shulyak 047c9b5263
Download transfers starting from latest block header () 2019-06-14 13:16:30 +03:00
Adam Babik c9e99c432d migrate to go 1.12 and go modules 2019-06-12 13:12:00 +02:00
Dmitry eeaf669006 Add goerli testnet 2019-03-19 10:56:09 +02:00
Dmitry Shulyak cf8f20b23c Use lumberjack library to rotate logs and add config that enables it () 2019-03-01 14:37:13 +01:00
Dmitry Shulyak aa28f652e3 Re-send user payload if previously sent envelope wasn't acknowledged ()
* Split shhext.tracker into envelopes and mail monitors

* Send envelopes on every new attempt to deliver a message

* Re-send user payloads if previous envelopes weren't acknowledged

* Remove debug api across the codebase
2019-03-01 14:36:21 +01:00
Adam Babik 8583eb2f37
fix TestSyncBetweenTwoMailServers flakiness 2019-02-22 10:24:06 +01:00
Ivan Daniluk 4f3f5ee574 Gomobile support () 2019-02-01 18:02:52 +01:00
Andrea Franz 929a5de757
Login with keycard ()
* add InjectChatAccount to account manager

* add InjectChatAccount to Backend

* add comments to LoginWithKeycard

* add LoginWithKeycard test to lib

* fix export comment to avoid lint errors

* ensure wallet and chat keys are empty before injecting chat key

* rename InjectChatAccount to SetChatAccount

* add TestBackendInjectChatAccount

* stop node in TestBackendInjectChatAccount

* SetChatAccount doesn't return error

* add comment to InjectChatAccount

* fix account test
2019-01-24 16:44:46 +01:00
Andrea Franz 4939268edf
Unlock wallet and chat keys ()
* select account decrypting wallet and chat keys

* adapt account tests to use chat and wallet account/keys

* fix tests using chat address

* changes after review

* fix status service e2e tests

* add account.Info struct returned when creating and recovering an account

* use s.EqualValues to compare recovered accounts

* return Info instead of *Info

* add both address and walletAddress to responses

* Update lib/types.go

Co-Authored-By: gravityblast <andrea@gravityblast.com>

* Update lib/types.go

Co-Authored-By: gravityblast <andrea@gravityblast.com>

* update comment to fix lint
2019-01-18 10:01:14 +01:00
Dmitry Shulyak 863d07f798 Accept status service config directly from json input () 2019-01-17 13:56:22 +01:00
Adam Babik fd48ad8b29
geth: upgrade geth to 1.8.21 () 2019-01-16 15:42:00 +01:00
Andrea Franz 06d4a99c09
add selected wallet and selected chat accounts ()
* use keyStore.ImportExtendedKeyForPurpose when creating keys

* rename selectecAccount to selectedWalletAccount

* add selectedChatAccount

* update e2e test to check that injected whisper key is the chat public key

* update TestSelectedAccountOnRestart to check that chat key is used in whisper

* use chat account in api/backend

* update mocks

* temporarily update VERSION to build a release

* check that chat/wallet keys are the same in a different test

* add account test to check that wallet and chat keys are the same

* test only that the chat key is injected in whisper

* put back the right VERSION
2019-01-09 09:47:06 +01:00
Adam Babik 08931fb761
Handle gracefully RPC calls when node is stopped () 2018-12-20 09:31:17 +01:00
Adam Babik 74cb16c456
add SyncMessages method to shhext api () 2018-12-14 12:21:34 +01:00
Adam Babik 5dbde3a94b
fix flaky TestSyncBetweenTwoMailServers () 2018-12-12 11:21:52 +01:00
Adam Babik 913dbfca9b
Support Mail Server data synchronization ()
These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
2018-12-06 10:48:28 +01:00
Dmitry Shulyak a609b468fe
Mail peer store and connection manager ()
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history

When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.

If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.

If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.

We will have two components that will rely on this logic:
1. requesting history

If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.

2. confirmation from selected mail server

Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.

I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server

Currently this feature is hidden, as certain changes will be necessary in status-react. 

partially implements: https://github.com/status-im/status-go/issues/1285
2018-12-05 15:57:05 +02:00
Adam Babik 52a1bdfed6
Upgrade geth 1.8.17 plus add metrics during compilation time ()
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.

The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
2018-11-14 08:03:58 +01:00
Dmitry Shulyak 3fe5b25ff2
Implement eth_getFilterLogs ()
Queries logs from remote server using original filter criteria.
2018-11-06 07:41:36 +01:00
Dmitry Shulyak 29b55bd445
Add local implementation of the newFilter call using remote getLogs call ()
* Implement subscriptions and filtering

* Add e2e test with log filter polling logs from EVM with clique backend

* Apply review comments

* Move devnode to t/devtests to avoid cycle in imports
2018-10-23 08:11:11 +03:00
Dmitry 5b551c67fa Use status-im whisper/whisperv6 module everywhere 2018-10-03 09:27:51 +03:00
Andrea Maria Piana dcaf8caed0
Add x3dh key exchange ()
* Add x3dh key exchange

* Encrypt using the double ratchet

* Multi device with auto-pairing

* Add pfs enabled flag
2018-09-24 20:07:34 +02:00
Adam Babik 8aef7c4f15
tune CLI for better docs experience () 2018-09-21 16:09:31 +02:00
Pedro Pombeiro 3d00af7fa3
Streamline configuration in status-go. Part of ()
- Replace command line flags with `-c` config flag. Part of 
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
2018-09-13 18:31:29 +02:00
Adam Babik 3521c2ac45
Remove NetworkID from cluster () 2018-08-26 12:54:58 +02:00
Kishan Sagathiya 0f5ad2545d Issue Read configuration without go-bindata ()
In the current codebase if you want to use different keys (or any
other configuration) for tests it is required to place them at a
certain path in static folder, bundle them with go-bindata and only
then run tests. This is simply unnecessary.

This commit instead directs loads configurations from file without
relying on go-bindata
2018-08-21 14:46:10 +02:00
Sebastian Delgado 4afd9e6c6c Remove transactions queue 1027 ()
Remove `PendingSignRequests` queue from the sign module.

This closes  by removing the pending sign requests queue dependency from the SendTransaction, SignMessage and Recover.
2018-08-16 13:37:53 +02:00
Pedro Pombeiro fb1e30c5f4 Disable NTP sync on the remaining e2e tests 2018-08-07 18:02:42 +02:00
Pedro Pombeiro 840b5b64d2 Implement mailserver canary service. Closes 2018-08-07 17:17:59 +02:00
Adam Babik 2bc3191fcf
Add custom handlers to the private RPC client () 2018-08-07 16:21:43 +02:00
Adam Babik 71b8e0e73d
Remove shh and shhext from APIModules ()
Addresses . By removing these two modules from APIModules, we disallow DApps from using dangerous methods like shh.SetMinPoW
2018-08-02 08:48:44 +02:00
Adam Babik 408ba5a7e6
Configure eth.staging fleet ()
Provide a way to run a status node with a different fleet.
2018-07-25 16:03:35 +02:00
Ivan Daniluk 801053a17d
Remove jail package () 2018-07-24 12:05:21 +02:00
Adam Babik 384567213c
fix flaky TestRequestMessageFromMailboxAsync test () 2018-07-20 08:34:38 +02:00
Adrià Cidre bfbb02019f
Discover mail servers on demand. ()
* [] Discover mail servers on start up.

* [] On-demand mail-server discovery
2018-07-16 09:40:40 +02:00
Andrea Franz 8ce522f430
remove old methods addPeerSync from tests () 2018-07-04 17:53:02 +02:00
Andrea Franz 6d1bcff446
use envelopeFeed to wait for envelopes to be available in tests ()
* remove sleeps in WhisperMailboxSuite

* use envelopeFeed to wait for envelopes to be available in group chat test

* remove unsed envelopeTracer in whisper e2e tests

* unsubscribe from envelopeFeed

* use s.addPeerSync instead of WaitForPeerAsync
2018-07-04 13:07:40 +02:00
Adam Babik 38a60135b2 Add asymmetric key support for MailServer requests ()
* add Asymmetric Key support for MailServer requests

* remove deprecated notice

* fix linter

* refactoring Whisper config related to MailServer

* fix race condition
2018-07-04 11:30:57 +02:00
Adrià Cidre f19e3eae8e
Move TestSentEnvelope to status-scale repo () 2018-07-02 22:29:03 +02:00
Andrea Franz 809db97e54
mailserver pagination ()
* mailserver sends envelopes in descending order

* add limit value in mailserver request payload

* mailserver sends messages up to the limit specified in the request

* update Archive method to return key and error

* processRequest returns the next page cursor

* add cursor to mailserver request

* add limit and cursor to request payload

* fix request limit encoding

* wait for request completed event in TrackerSuite/TestRequestCompleted

* add cursor to mailserver response

* fix cursor position in payload

* add e2e test for mail server pagination

* validate mail server response size

* remove old limitReached var

* fix lint warnings

* add whisper patch

* fix tests after rebase

* check all return values to avoid lint warnings

* check that all messages have been retrieved after 2 paginated requests

* fix lint warnings

* rename geth patch

* merge mailserver patches into one

* add last envelope hash to mailserver response and EventEnvelopeAvailable event

* update whisper patch

* add docs to MailServerResponse

* update whisper patch

* fix tests and lint warnings

* send mailserver response data on EventMailServerRequestCompleted signal

* update tracker tests

* optimise pagination test waiting for mailserver to archive only before requesting

* rollback mailserver interface changes

* refactoring and docs changes

* fix payload size check to determine if a limit is specified

* add more docs to the processRequest method

* add constants for request payload field lengths

* add const noLimits to specify that limit=0 means no limits
2018-07-02 09:38:10 +02:00
Adrià Cidre 3ca120c2aa
[] : Introducing t/helpers and WaitForPeer ()
* [] : Introducing t/helpers and WaitForPeer

* waitForPeer to receive a p2p.Server
2018-06-27 09:55:25 +02:00
Pedro Pombeiro ab7b9b914c Recover from panics which can get sent from goleveldb due to bad arguments. Closes 2018-06-26 16:49:13 +02:00
Adrià Cidre 5bbfabde94
New debug_postSync endpoint ()
* [#ISSUE-1041] New debug_postconfirm endpoint

* move debug service api inside shhext service
2018-06-25 15:27:17 +02:00
Pedro Pombeiro 58a8135a87 Fix return type of RequestMessages (hash instead of bool) 2018-06-20 18:17:12 +02:00
Adrià Cidre 56cc3d20da
First step to remove StatusAPI () 2018-06-19 09:49:24 +02:00
Pedro Pombeiro 2b290d1a76 Address PR review comments 2018-06-18 15:24:27 +02:00
Pedro Pombeiro fac5576988 Use `EnvelopeTracer` as an event source to avoid relying on timeouts in tests. Fixes
Fix bug in mailserver implementation which was using upper bound `to` parameter as an exclusive, rather than inclusive parameter, as stated in the documentation: https://github.com/status-im/status-go/wiki/Additional-JSON-RPC-API
2018-06-18 15:24:27 +02:00
Andrea Franz fa390a52ae
RequestHistoricMessages response ()
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Pedro Pombeiro 354e23aaf5 Fix issues reported by lint. Part of 2018-06-14 13:52:51 +02:00
Adrià Cidre 6173d3ee06
[] Make ntp synchronization configuratble () 2018-06-13 13:24:04 +02:00
Adam Babik bbe7fba0c9
Benchmark tests ()
A set of tests used to benchmark Whisper and MailServer nodes.
2018-06-11 12:48:42 +02:00
Pedro Pombeiro 291ad8fa35 Fail tests if `ACCOUNT_PASSWORD` env var is not defined 2018-06-08 16:40:17 +02:00
Adrià Cidre d5be8c525d
[] move geth subpackages to root level () 2018-06-08 13:29:50 +02:00
Pedro Pombeiro feed9158bd Temporarily revert "Remove shh and shhext from APIModules and update tests ()"
This reverts commit 7581766470.
2018-06-07 18:18:40 +02:00
Sebastian Delgado 7581766470
Remove shh and shhext from APIModules and update tests () 2018-06-04 13:13:02 -04:00
Adam Babik 717d0fbadb
Fix another round of flaky tests () 2018-05-29 13:24:23 +02:00
Adrià Cidre cf8ad7e002
[#ISSUE-947/ISSUE-946] Rate limit and batch limit on any mail server query () 2018-05-17 13:21:04 +02:00
Dmitry Shulyak 5aae87aba8
Prevent sending messages to flaky peers () 2018-05-15 20:08:31 +03:00
Igor Mandrigin a933885806 Deduplicator: add API to confirm received messages. 2018-05-10 15:28:48 +02:00
Adrià Cidre 37a58a513d [] : False positive e2e test fix 2018-05-10 15:18:37 +02:00
Igor Mandrigin 397bf83924 Add testcases for contract mining to the jail RPC tests. 2018-05-09 14:34:10 +02:00
Igor Mandrigin 31cf2297d2 Implement block filters API for the RPC mode.
Some operations (like deploying contracts) require filter APIs to work.
Since these operations aren't supported on Infura anymore, and we don't
run LES, a separate implemenation of filters is required.

Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
2018-05-08 19:46:25 +02:00
Dmitry Shulyak 707221954f
Make whisper tolerant to local time skews ()
This change adds adds an ability to use different source of time for whisper:

when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.

So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.

In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.

Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.

Closes: 
2018-05-04 11:23:38 +03:00
Dmitry Shulyak 357786eeca Fix bloom filter expectations in group chat test
Every peer must be subscribed to the topic that is used to send messages.
In the test Alice was communicating with Bob and Charlie over custom topic, but
that topic wasn't added to a bloom filter, thus a certain flake was possible.

Normally it wasn't causing problems because syncAllowance in whisper, which is 10s:
- we set bloom filter to all zeros
- but we still will accept all envelopes for 10s
- in case we send first envelope into such channel after sync allowance - we will get an error such
that envelope doesn't match the bloom filter
2018-05-04 10:09:27 +03:00
Adrià Cidre 8c9db81bec
Expose status specific methods (login/signup/joinpublicchannel) through the RPC api () 2018-05-03 12:36:56 +02:00
Ivan Danyliuk 953c26e8cf
Move signal logic into signal package 2018-05-03 09:35:58 +02:00
Frank Mueller c85e0802eb
Remove mention of DevMode ()
* Remove mention of DevMode

* Remove prodMode
2018-04-26 19:59:57 +02:00
Andrea Franz 5f45f6498d
Add LogEnabled attribute to NodeConfig to, used in the call from status-react ()
* add LogEnabled attribute to NodeConfig, used in the call from status-react

* fix use of OverrideRootLogin in tests

* enable logger in tests based on LogLevel

* move LogEnabled before LogFile
2018-04-26 18:28:42 +02:00
Frank Mueller b543d32a31
Move ssh_requestMessages to sshext_requestMessages ()
* Add RequestMessage to sshext

* E2E tests now use shhext_requestMessages

* Typo in comment

* Enhanced maintainability

* Drop former mailservice

* Code reorg after review

* Fix missed changes after update to 1.8.5
2018-04-26 07:56:19 +02:00
Dmitry Shulyak f3e2631c1d
Add signals for notifying react about discovery activity ()
* Add signal for notifying react on discovery activity

* Whitelist expected signals in e2e tests
2018-04-25 10:13:59 +03:00
Dmitry Shulyak b37fda7731
Update go-ethereum to 1.8.5 ()
* Rebase on 1.8.5

* Remove outdated patches and apply all others

* Use shh_post that returns hash

* Use bloom filter for request to mailserver

* Remove tests for sending messages without subbing first

* Fix deadlock in ethdb

* Expect null if receipt is not yet created

* Subscribe to messages before sending them in whisper test
2018-04-24 18:50:26 +03:00
Adam Babik b85e50cbc9
Remove any logic from api.go [breaking-change] ()
Deprecated `Notify` binding got removed.
2018-04-23 15:34:49 +02:00
Adam Babik 5f075eeaba
Enable logs in tests ()
Pass `-log` option to `go test` in order to change the log level. By default, it's `INFO`.
2018-04-22 18:50:34 +02:00
Adam Babik 07b2c3d750
Move concurrent backend test from e2e to unit () 2018-04-20 17:39:53 +02:00
Dmitry Shulyak 2f2dfe16c0
Add default peer limits configuration ()
* Add default peer limits configuration

If discovery is enabled for a given cluster - we will set a default
expected number of peers for each enabled service. For example:

- if cluster is rinkeby has a discovery enabled we will
  check which services are enabled
- if whisper is enabled we will set min and max limits by default
- if les is enabled and infura is not used we will set limits too

When statusd is used - configuration must be provided using configuration
supported by statusd.


* Fix deadlock in les peer set
2018-04-20 15:23:18 +03:00
Igor Mandrigin f0f55d408f Make sure that only `personal_sign` and `personal_ecRecover` are exported. 2018-04-18 18:22:21 +02:00
Frank Mueller 6598371dc0
Prepare tests for mainnet ()
* Start enabling to test Mainnet

* Minor corrections found during code scan

* Set mainnet blocker in E2E again

* Introduced securing of mainnet transaction tests

* Fix typing error

* Typo led to follow-up errors

* Linter problem

* Change to individual test skips after review

* More flexible skip control for mainnet and status chain

* Fix double space

* Change of skipping method and fixing wrong skipped networks
2018-04-18 17:13:27 +02:00
Igor Mandrigin 19556554a6 Fix dataraces in e2e/transactions tests 2018-04-18 15:02:59 +02:00
Igor Mandrigin 4cc6028d59 Implement `personal_sign`.
This commit implements `personal_sign` RPC or web3 personal.sign
methods.

NB! Contains breaking API changes.
2018-04-17 11:18:58 +02:00
Adam Babik 364f88efd9
Allow to start ephemeral StatusNode ()
This change will greatly simplify writing unit tests when a node is required but data persistence is irrelevant.

I also Introduced some refactoring and unit tests for `StatusNode`.
2018-04-16 14:36:09 +02:00
Dmitry Shulyak 0b123ed407 Send an expiration signal when envelope wasn't delivered to any peer 2018-04-15 10:44:49 +03:00
Adam Babik 0d652c3851
Expose only public APIs in inproc RPC client [breaking-change] ()
`CallRPC` binding, which is used as a provider for web3.js, exposes only public or whitelisted APIs.
2018-04-12 18:17:10 +02:00
Dmitry Shulyak ba9a25e284
Implement shh api extension that allows to confirm that message is sent ()
* Implement shh api extension that allows to confirm that message is sent

* Add a patch

* Fix linter

* Add readme

* Add tests for tracker

* Address review
2018-04-11 18:41:51 +03:00
Igor Mandrigin a9eb5a7d2b Generalize signing requests.
We need to be able to sign more than just transactions to make DApps
work properly. This change separates signing requests from the
transactions and make it more general to prepare to intoduce different
types of signing requests.

This change is designed to preserve status APIs, so it is
backward-comparible with the current API bindings.
2018-04-09 20:48:00 +02:00
Adam Babik 32d71da046
fix linter in transactions_test.go () 2018-04-09 14:14:02 +02:00
Dmitry Shulyak 4d00fa80b0 Ensure that To field is nil when contract is created 2018-04-09 14:41:50 +03:00
Adrià Cidre df35ad6dbe
[] Unify EnsureSync on t/utils/utils.go and StatusManager () 2018-04-09 09:16:43 +02:00
Adrià Cidre 359b3621e9 [] Simplify node.Manager public api and rename it to StatusNode
- [x] [] : Remove unused methods PopulateStaticPeers, ReconnectStaticPeers, removeStaticPeers, removePeer
- [x] [] : Rename node.Manager to node. StatusNode and simplify its public api
- [x] [] : Rename all references to nodeManager to statusNode
2018-04-05 16:47:27 +02:00
Adrià Cidre aa0f2ede6d
[] : Remove geth/common package () 2018-04-04 19:39:38 +02:00
Adrià Cidre 518148cab9
Merge pull request from status-im/improvement/issue-779/common-NodeManager-removal
[] common.NodeManager removal
2018-03-29 09:57:18 +02:00
Igor Mandrigin 3b32ab8426 Support both "input" and "data" fields when creating a transaction. 2018-03-28 20:28:35 +02:00
Adrià Cidre 1b724a81a5
[] common.NodeManager removal 2018-03-28 17:39:09 +02:00
Adrià Cidre e646001578
Merge pull request from status-im/transactions-and-accountmanager
[] txQueueManager to not depend on AccountManager and nodeManager
2018-03-28 11:06:33 +02:00
Adrià Cidre c76b2a0f62 [] : Split accounts and whisper tests () 2018-03-28 10:24:55 +02:00
Adrià Cidre 13cb2daaa2
[] txQueueManager to not depend on AccountManager 2018-03-28 09:58:05 +02:00
Evgeny Danienko 9ccd9b20e8
Merge branch 'develop' into 2018-03-27 18:35:58 +03:00
Pedro Pombeiro d9b28a06d9
Take into consideration that path may contain multiple `status-go` directories, take the last ()
- Happens in Jenkins
2018-03-27 17:06:36 +02:00
Pedro Pombeiro c4d7a6167e
Add `race-check` target to Makefile. Closes ()
* Add `race-check` and `test-unit-race` target to Makefile. Closes 
2018-03-27 14:38:41 +02:00
Evgeny Danienko b9598a1a31
led comments to the standard 2018-03-26 20:05:24 +03:00
Evgeny Danienko a90be3c0de
fixes after code review 2018-03-26 19:54:27 +03:00
Evgeny Danienko aaf2b41e2e
test sending message without subscription 2018-03-26 19:02:02 +03:00
Adrià Cidre 18fbebc942
[#issue-403] Account package cleanup 2018-03-23 08:21:40 +01:00
Alex Kohler 4a01a395b4 Move status-go over to go-ethereum/log () 2018-03-20 19:35:28 +01:00
Frank Mueller 8e42014671
Static peers are now also optionally configurable ()
* Rename bootnode to static peers

Also add some groupings between types.

* Remove not needed genesisHash

* More cleanup of bootnodes with static peers

* Add option of cluster configuration file

* New generated bindata.go

* Changes after npm install

* Add argument for cluster configuration file

* Add test for dynamic cluster loading

Not yet sure with name "cluster config".

* Solved conflicts

* Renaming of static peers

* Remove static peers population

* Missing argument for config

* Renaming of static peers to boot nodes for consistency

* Fix of name change

* Cluster config is now cluster data

* Load static nodes from configuration

* Final renaming of var for file content
2018-03-19 17:22:09 +01:00
Frank Mueller 81c8669518
Code review for Mainnet ()
* Add debug log to test package t

* Add comments for test restrictions

* Add explicit errors for mainnet

* Panic instead of error in case of mainnet during tests

* Now panic with illegal network

* Validate the compiler flag for mainnet

* Made test for unset mainnet explicit

* Forgot to use my flag

* Linter has troubles with unused field

* Add default network id

In e2e tests there's no default anymore in go code. So ensure in Makefile.

* Fix dumb syntax error

* CI needs build with Mainnet

* Explicit rebuild only for tests

* No more separation of mainnet and testnet builds

* Build flags now set in Makefile
2018-03-15 12:36:28 +01:00
Dmitry Shulyak a5cec358a9 Post whisper message asynchronously in tests
I am not 100% percent sure what is happening but it seems that
newMessageFilter is async operation, result of this operation is assumed
to be used in callbacks. All other tests are doing at least 1 io operation
in between creating a filter and posting a message, and it must be enough
for newMessageFilter to complete.

Setting higher GOMAXPROCS allows Otto vm to execute received io requests
immediatly. While lower number of processes may result in events re-ordering.

https://github.com/ethereum/web3.js/blob/develop/lib/web3/methods/shh.js#L39-L41

I tested this change with GOMAXPROCS=1 and it passes consistently.
2018-03-02 19:43:08 +01:00
Pedro Pombeiro 6cdea4ef97
Update codebase to leverage Whisper v6 ()
* Update project to use Whisper v6. Part of 

* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.

* Enable light mode on Whisper v6 for non-mail servers. Part of 

* Fix race condition in whisperv6/peer.go. Part of  (PR already accepted upstream for 1.8.2)

* Update bootnode addresses in staticnodes.json. Part of 

* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of 

* Move MakeTestNodeConfig to utils. Part of 

* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of 
2018-03-02 10:25:30 +01:00
Alex Kohler 5fbacb49e1 Remove GetObjectValue from JailCell interface () 2018-03-01 17:48:30 +01:00
Pedro Pombeiro e4cbce12c4
Update to go-ethereum 1.8.1 ()
* Update `github.com/ethereum/go-ethereum` package to 1.8.1 branch. Part of 
* Fix code due to some signature changes. Part of 
* use upstream for whisper backend
* Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of 
* Take into account the DNS rebinding protection introduced in 1.8.0 by adding exception for localhost. Part of 
* Add patches required for cross-compiled builds starting with geth 1.8.0. Only applied during build. Part of 
* Update expected JSON result in `TestRegressionGetTransactionReceipt()` and `TestCallRawResultGetTransactionReceipt()`. Part of 
* Fix some failing e2e tests. Part of 
* Address comments in PR . Part of 
2018-02-27 11:39:30 +01:00
Pedro Pombeiro 0332465952
Update `make generate` target to account for new `_assets` folder 2018-02-26 16:04:40 +01:00
Dmitry Shulyak 741422af73
Add a test with network connection problem while syncing a chain ()
Test waits till synchronization is started, then interrupts network connection, wait for a failure event and restore network connection, confirming that after the connection is restored synchronization will proceed.
2018-02-23 09:03:55 +02:00
Igor Mandrigin c06d58addd Remove `notifications` package from go-ethereum. 2018-02-22 11:10:21 +01:00
Pedro Pombeiro 2993dcc3db
Fix a couple race conditions in tests ()
* Fix race condition on `LoopSuite`
* Fix race condition in the access to a Otto.Value in tests (quick fix, need to improve approach in the future)
2018-02-21 13:18:32 +01:00
Dmitry Shulyak b2580c79d7
Add destructive tests ()
Network disconnect is introduced by removing default gateway, easily reversible condition.

On my local machine it takes 30 seconds for peers to reconnect after connectivity is restored. As you guess this is not an accident, and there is 30 seconds timeout for dial expiration. This dial expiration is used in p2p.Server to guarantee that peers are not dialed too often.

Additionally I added small script to Makefile to run such tests in docker environment, usage example:

```
make docker-test ARGS="./t/destructive/ -v -network=4"
```
2018-02-19 17:32:58 +02:00
Pedro Pombeiro 7410532879 Patcher improvement ()
Improve patcher tool:
* fix broken `-p` flag,
* rename it to `-b`,
* add a new `-p` flag that allows specifying the source of the patches, so that it can be used for more than just go-ethereum patches,
* make it work correctly independently of the order that options are specified in the command line.
2018-02-18 09:14:27 -07:00
Dmitry Shulyak 4a49e840e3 Remove status backend from les internal api () 2018-02-16 15:40:37 +01:00
Dmitry Shulyak 0b979c507d Fix async status api () 2018-02-14 17:32:36 +01:00
Adam Babik adbc7b5ae6 use s.Require() in Whisper tests 2018-02-14 10:45:29 +01:00
Alex Kohler 365bc662a2 Enable gometalinter on tests and fix static analysis issues ()
* Enable gometalinter on tests and fix static analysis issues

* Remove unneeded change

* Fix additional lint errors

* Add nolint directives and error checks

* Add error assertions instead of nolint directives

* Go back to using lint directive for loop.Run goroutine

* Add error check to loop.Run
2018-02-12 13:16:06 +02:00
Dmitry Shulyak 2d964bfe9f Remove async operations from node manager ()
The main goal of this change is to remove async operations from node manager.
Additionally all of the signals from node manager are moved to status backend.

All of the async operation now will have the following behaviour:
- If node in the correct state exit immediatly without error
- If node not in the correct state exit immediatly with error
- In all other cases spawn a goroutine with wanted operation
- All the progress regarding that operation will be reported
  by using signals
- Signals should be handled in once place, which is StatusBackend

There are 2 potentially breaking changes:
- Empty event field will be ommited when Envelope is sent to a client
- All errors will be delivered to a client as an Envelope, previously
  some errors (NodeExists, NoRunningNode) were delivered synchronously

Signed-off-by: Dmitry Shulyak <yashulyak@gmail.com>
2018-02-09 14:37:56 +01:00
Sebastian Klier 70912ab8f6 Group tests in a dedicated testing package () 2018-02-08 13:52:47 +01:00