Commit Graph

207 Commits

Author SHA1 Message Date
Andrea Maria Piana c88df6c1c7
Handle messages sent to all devices 2020-07-27 08:51:09 +02:00
Andrea Maria Piana 7e8d1353d0
Create server identity on start if necessary 2020-07-27 08:51:08 +02:00
Andrea Maria Piana d985af4a7e
Implement server persistence for client & basic tests 2020-07-27 08:51:01 +02:00
Andrea Maria Piana 9da64ef251
add basic config 2020-07-27 08:50:59 +02:00
Andrea Maria Piana c4fa9825a9
Mute chat 2020-07-27 08:50:46 +02:00
Andrea Maria Piana 4720224ba2 expose getContactByID method 2020-06-17 12:24:58 +02:00
Andrea Maria Piana d5086d6e89 Remove incentivisation service
Incentivisation was an experiment in running an incentivised fleet that
rewarded nodes based on their well behavior. It was heavily influenced
by https://docs.loki.network/ . It is currently not used anymore, so
removing.
2020-06-01 11:30:04 +02:00
André Medeiros be683556ff
fix: close resultsets so we don't leak them (#1809)
* fix: close resultsets so we don't leak them

* Refactor browsers/database

To implement PR suggestions and improve code quality.

* Refactor services/permissions/database

To implement PR suggestions and improve code quality.

Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2020-05-14 11:51:32 +01:00
Andrea Maria Piana 0e27e464d3 Add tests for sendBundle and fix flaky tests 2020-05-01 10:14:04 +02:00
Andrea Maria Piana aa7f591587 Move networking code for waku under `v0` namespace
Why make the change?

As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.

This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .

All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.

In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.

This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.

What has changed?

- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`

Things still to do

Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.

Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
2020-04-27 14:58:02 +02:00
Andrea Maria Piana 4d2fb67add Allow to leave chat without removing it 2020-04-14 14:08:54 +02:00
Andrea Maria Piana f3fc6812cd Add change group chat name endpoint 2020-04-14 14:08:54 +02:00
Andrea Maria Piana 3c5354280c Return count of updated messages in MarkMessagesSeen
The frontend needs to know if the messaged marked as seen was actually
seen or not, as we might not have it loaded in the frontend.
2020-04-07 13:23:21 +02:00
Andrea Maria Piana fd4c627c2e Ignore full node flag when in topic mode 2020-03-23 15:10:07 +01:00
Andrea Maria Piana 9ee0e52ea3 Ignore full node flag when in topic mode 2020-03-23 15:10:07 +01:00
Roman Volosovskyi 9d5684162a
Stop fetching new blocks while app is not active
Saves some traffic (blocks with transactions are heavy) and infura
requests.
2020-03-11 08:39:41 +02:00
Adam Babik 831ab85561
Upgrade deps (#1892) 2020-03-09 08:55:58 +01:00
Andrea Maria Piana 5cc60de80b
Add mark as read endpoint (#1863) 2020-02-26 13:31:48 +01:00
Andrea Maria Piana 7a64fbf5f7
make apis private (#1866)
* make apis private

* Bump whisper
2020-02-25 08:03:01 +01:00
Jakub Sokołowski 86f54c3f5f remove eth.beta fleet as it has been decomissioned
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-02-21 14:48:47 +01:00
Adam Babik 76b5dc29dc
Integrate Whisper-Waku bridge in status-go (#1854) 2020-02-18 12:21:01 +01:00
Adam Babik 491e3be799
fix flaky tests (#1860) 2020-02-18 10:34:09 +01:00
Andrea Maria Piana 141f917e7c
Initialize protocol when waku active (#1832) 2020-02-17 15:38:59 +01:00
Jakub Sokołowski cca47ef6e1 rename gomarkdown/markdown to status-im/markdown
This resolves a dependency conflict we have with MatterBridge
which was using a newer version of the same package.
This resulted in a JSON marshalling bug that would crash the bridge.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-02-11 16:06:43 +01:00
Adam Babik 8b61d92d58
Remove protocol and eth-node submodules (#1835) 2020-02-10 12:22:37 +01:00
Roman Volosovskyi dc80cb0e06
Cleanup after tx fetching changes
- unused API methods are removed
- some unusued code is removed too
- API docs are updated

That's just a portion of clean up that should be done,
but the rest of it will probably happen in different PR
with changes to the way how we watch to chain updates.
2020-02-09 12:05:36 +02:00
Andrea Maria Piana f2eebd1e49
Verify ENS in the background (#1824)
Currently ENS are verified explicitly by status-react, this is not ideal
as if that fails it will have to be explicilty retried in status-react.
This commits changes that behavior so that ENS are verified in a loop
and updated if new messages are received.
2020-02-05 11:09:33 +01:00
Roman Volosovskyi c2f22f1fbc
[status-im/status-react#9927] Fast blocks sync after delay
- In order to avoid handling of the reorganized blocks we use an offset
from the latest known block when start listening to new blocks. Before
this commit the offset was 15 blocks for all networks. This offset is
too big for mainnet and causes noticeable delay of marking a transfer as
confirmed in Status (comparing to etherscan). So it was changed to be 5
blocks on mainnet and is still 15 blocks on other networks.
- Also before this commit all new blocks were handled one by one with
network specific interval (10s for mainnet), which means that in case of
lost internet connection or application suspension (happens on iOS)
receiving of new blocks would be paused and then resumed with the same
"speed" - 1 blocks per 10s. In case if that pause is big enough the
application would never catch up with the latest block in the network,
and this also causes the state of transfers to be delayed in the
application. In this commit in case if there was more than 40s delay
after receiving of the previous block the whole history in range between
the previous received block and ("latest"-reorgeSafetyDepth) block is
checked at once and app catches up with a recent state of the chain.
2020-01-30 17:25:56 +02:00
Andrea Maria Piana 8931b14c4e
Explicitly init messenger (#1821)
Messenger needs to wait until all the topics/mailservers are loaded in
the client, so we explicitly add a way to call Init.
2020-01-29 20:40:06 +01:00
Roman Volosovskyi a92a95cf83
status-im/status-react#9203 Faster tx fetching with less request
*** How it worked before this PR on multiaccount creation:
- On multiacc creation we scanned chain for eth and erc20 transfers. For
  each address of a new empty multiaccount this scan required
  1. two `eth_getBalance` requests to find out that there is no any
     balance change between zero and the last block, for eth transfers
  2. and `chain-size/100000` (currently ~100) `eth_getLogs` requests,
     for erc20 transfers
- For some reason we scanned an address of the chat account as well, and
  also accounts were not deduplicated. So even for an empty multiacc we
  scanned chain twice for each chat and main wallet addresses, in result
  app had to execute about 400 requests.
- As mentioned above, `eth_getBalance` requests were used to check if
  there were any eth transfers, and that caused empty history in case
  if user already used all available eth (so that both zero and latest
  blocks show 0 eth for an address). There might have been transactions
  but we wouldn't fetch/show them.
- There was no upper limit for the number of rpc requests during the
  scan, so it could require indefinite number of requests; the scanning
  algorithm was written so that we persisted the whole history of
  transactions or tried to scan form the beginning again in case of
  failure, giving up only after 10 minutes of failures. In result
  addresses with sufficient number of transactions would never be fully
  scanned and during these 10 minutes app could use gigabytes of
  internet data.
- Failures were caused by `eth_getBlockByNumber`/`eth_getBlockByHash`
  requests. These requests return significantly bigger responses than
  `eth_getBalance`/`eth_transactionsCount` and it is likely that
  execution of thousands of them in parallel caused failures for
  accounts with hundreds of transactions. Even for an account with 12k
  we could successfully determine blocks with transaction in a few
  minutes using `eth_getBalance` requests, but `eth_getBlock...`
  couldn't be processed for this acc.
- There was no caching for for `eth_getBalance` requests, and this
  caused in average 3-4 times more such requests than is needed.

*** How it works now on multiaccount creation:
- On multiacc creation we scan chain for last ~30 eth transactions and
  then check erc20 in the range where these eth transactions were found.
  For an empty address in multiacc this means:
  1. two `eth_getBalance` transactions to determine that there was no
     balance change between zero and the last block; two
     `eth_transactionsCount` requests to determine there are no outgoing
     transactions for this address; total 4 requests for eth transfers
  2. 20 `eth_getLogs` for erc20 transfers. This number can be lowered,
     but that's not a big deal
- Deduplication of addresses is added and also we don't scan chat
  account, so a new multiacc requires ~25 (we also request latest block
  number and probably execute a few other calls) request to determine
  that multiacc is empty (comparing to ~400 before)
- In case if address contains transactions we:
  1. determine the range which contains 20-25 outgoing eth/erc20
     transactions. This usually requires up to 10 `eth_transactionCount`
     requests
  2. then we scan chain for eth transfers using `eth_getBalance` and
     `eth_transactionCount` (for double checking zero balances)
  3. we make sure that we do not scan db for more than 30 blocks with
     transfers. That's important for accounts with mostly incoming
     transactions, because the range found on the first step might
     contain any number of incoming transfers, but only 20-25 outgoing
     transactions
  4. when we found ~30 blocks in a given range, we update initial
     range `from` block using the oldest found block
  5. and now we scan db for erc20transfers using `eth_getLogs`
     `oldest-found-eth-block`-`latest-block`, we make not more than 20 calls
  6. when all blocks which contain incoming/outgoing transfers for a
     given address are found, we save these blocks to db and mark that
     transfers from these blocks are still to be fetched
  7. Then we select latest ~30 (the number can be adjusted) blocks from
     these which were found and fetch transfers, this requires 3-4
     requests per transfer.
  8. we persist scanned range so that we know were to start next time
  9. we dispatch an event which tells client that transactions are found
  10. client fetches latest 20 transfers
- when user presses "fetch more" button we check if app's db contains next
  20 transfers, if not we scan chain again and return transfers after

small fixes
2020-01-23 10:36:11 +02:00
Adam Babik 79b8112f89
Split shhext into shhext and wakuext (#1803) 2020-01-20 21:56:06 +01:00
Pedro Pombeiro db01f0b3e4 Avoid passing node to subscriptions service 2020-01-20 13:15:17 +01:00
Pedro Pombeiro 6537cae606 Nimbus node support 2020-01-20 13:15:17 +01:00
Andrea Maria Piana 25d46c6d82
Fix waku tests & contact ens (#1802) 2020-01-17 13:39:09 +01:00
Andrea Maria Piana c569d8a4ed
Sync installation messages & contact requests (#1791) 2020-01-15 08:25:09 +01:00
Adam Babik 44aa313981
Make shhext and protocol work with Waku (#1777)
This change makes shhext and protocol submodule work with Waku and Whisper.
2020-01-13 20:17:30 +01:00
Andrea Maria Piana 88a1d0111e
Add Commands (#1731)
This commit adds handling of Transaction commands.
2020-01-10 19:59:01 +01:00
Adam Babik 37a508a97b
Integrate Waku service (#1759) 2020-01-08 12:12:23 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 527cd38e1f Address code review comments 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 287e5cdf79 Abstract `accounts.Key` and geth `keystore` 2020-01-06 10:17:23 +01:00
Adam Babik 1ac515f19e remove reSelectAccount from backend, SelectKeyPair and SelectedKeyPairID methods from Whisper 2019-12-30 12:56:05 +01:00
yenda 024f30f0b9 Put settings in a table with schema (#1746)
Move settings table schema from a key-value store to a one row table with many columns.

We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.

Co-authored-by: Adam Babik <a.babik@designfortress.com>
2019-12-27 10:58:25 +01:00
Pedro Pombeiro dd894ece15 Start abstracting geth Keystore 2019-12-19 14:11:48 +01:00
Pedro Pombeiro 7f11030896 Implement AddPeer/RemovePeer on eth-node 2019-12-19 14:11:48 +01:00
Pedro Pombeiro 41a6502340 Remove `SelectKeyPair` from `eth-nodes/types` 2019-12-19 14:11:48 +01:00
yenda f855228010 add accounts_deleteAccount method (#1738)
* add accounts_deleteAccount method

* set account created and updated at dates, order by creation date
2019-12-16 10:23:36 -05:00
Adam Babik 52dd835692
Implement rate limits exchange (#1729) 2019-12-11 14:11:29 +01:00
Adam Babik 203f29b13e
Fix race condition in subscriptions (#1646) 2019-12-11 09:44:57 +01:00
André Medeiros 0571f561f0
APIs to Get and Create custom Tokens (#1717) 2019-12-10 12:31:08 -05:00