Commit Graph

230 Commits

Author SHA1 Message Date
Andrea Maria Piana 0b2bd2863b Log mailserver error
while debugging I noticed we don't log `err` if the query fails.
2020-12-07 15:03:06 +01:00
Jonathan Rainville b2d8f7f416
feat: add bookmarks table for the browser bookmarks
plus get bookmark icon using a lib
2020-11-16 13:31:34 -05:00
Jakub Sokołowski 1e3801d105 use types.BytesToHash to print []byte variables
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-06-04 09:45:47 +02:00
Andrea Maria Piana aa7f591587 Move networking code for waku under `v0` namespace
Why make the change?

As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.

This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .

All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.

In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.

This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.

What has changed?

- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`

Things still to do

Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.

Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
2020-04-27 14:58:02 +02:00
Jakub Sokołowski c8d20f84f8 set archived_envelopes_total on start based on DB count
Before the count was reset every time process was restarted.
Also add 'db' label to distinguish between waku and whisper DBs.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-04-09 13:08:14 +02:00
Andrea Maria Piana 849492fda9
Add statusUpdate code ()
* Add status-option code

This commits changes the behavior of waku introducing a new status-code,
`2`, that replaces the current single options codes.

* linting
2020-02-21 15:48:53 +01:00
Volodymyr Kozieiev 19fc448c85
pending-stickers field added to settings, unit-tests fixed () 2020-02-13 13:48:04 +02:00
Jakub Sokołowski cca47ef6e1 rename gomarkdown/markdown to status-im/markdown
This resolves a dependency conflict we have with MatterBridge
which was using a newer version of the same package.
This resulted in a JSON marshalling bug that would crash the bridge.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-02-11 16:06:43 +01:00
Adam Babik bc2d018483
Add support for request messages by topics () 2020-01-21 08:11:24 +01:00
Adam Babik 87f1f8a965
Remove 24h time range validation from maislerver request () 2020-01-14 08:16:35 +01:00
Adam Babik 179e3e851e
Report correct number of pruned rows in Postgres mailserver () 2020-01-13 21:13:28 +01:00
Adam Babik 44aa313981
Make shhext and protocol work with Waku ()
This change makes shhext and protocol submodule work with Waku and Whisper.
2020-01-13 20:17:30 +01:00
Andrea Maria Piana 88a1d0111e
Add Commands ()
This commit adds handling of Transaction commands.
2020-01-10 19:59:01 +01:00
Adam Babik 37a508a97b
Integrate Waku service () 2020-01-08 12:12:23 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
yenda 024f30f0b9 Put settings in a table with schema ()
Move settings table schema from a key-value store to a one row table with many columns.

We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.

Co-authored-by: Adam Babik <a.babik@designfortress.com>
2019-12-27 10:58:25 +01:00
André Medeiros 0571f561f0
APIs to Get and Create custom Tokens () 2019-12-10 12:31:08 -05:00
Adam Babik a636f33109
Set Whisper version submodule to v6 () 2019-12-09 11:36:14 +01:00
Adam Babik 4ac4a61e20
Move whisper to status-go monorepo () 2019-12-09 11:06:04 +01:00
Andrea Maria Piana fd49b0140e
Move to protobuf for Message type ()
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`

* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.

* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.

* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
2019-12-05 17:25:34 +01:00
Pedro Pombeiro 26b6d7c36a Create a home submodule for Eth node bridges- Rename StatusBackend to GethStatusBackend 2019-11-27 17:02:09 +01:00
Pedro Pombeiro ed5a5c154d
Move to monorepo structure ()
Move to a monorepo structure with submodules

- Rename status-protocol-go to status-go/protocol
2019-11-21 17:19:22 +01:00
Pedro Pombeiro 2dd74da23d Integrate Nimbus status-protocol-go 2019-11-18 14:30:03 +01:00
Corey 457acec72e
added name to batch loop so break works properly () 2019-11-06 09:12:05 -05:00
Adam Babik 183d626256
Implement mailserver Deliver() with MessagesRequest ()
The main difference is that it takes `whisper.MessagesRequest` as an argument instead of `whisper.Envelope`.
2019-11-04 21:15:21 +01:00
Jakub Sokołowski c1f17cb4a3 use prometheus/client_golang for metrics
also re-implement mailserver metrics

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-11-04 13:18:50 +01:00
Adam Babik c74c36742e
Log error in mailserver () 2019-10-15 16:33:58 +02:00
Andrea Maria Piana b27779aa4e
Add Mailserver Request Gaps service ()
This commits add some endpoints to store mailserver request gaps,
and provides 4 endpoints:

```
AddMailserverRequestGaps(gaps []MailserverRequestGap) error
GetMailserverRequestGaps(chatID string) []MailserverRequestGap
DeleteMailserverRequestGaps(ids []string) error
DeleteMailserverRequestGapsByChatID(id string) error
```
2019-09-04 12:04:17 +02:00
Adam Babik d7f454fe4c
Add mailservers service () 2019-09-04 08:25:33 +02:00
Adam Babik 1ab2e88bf5
Sync messages in a loop until error or no cursor () 2019-06-26 18:17:41 +02:00
Andrea Maria Piana cef7f367ab Add topic negotiation
This commit add topic negotiation to the protocol.

On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.

This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).

When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.

This change is backward compatible, as long as there is no downgrade of
the app on the other side.

A few changes:

* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).

* Factored out multidevice management in a separate namespace

* Moved all the test to test the whole protoocl rather than just the encryption service

* Moved all the filter management in status-go
2019-06-19 17:38:45 +02:00
Adam Babik c9e99c432d migrate to go 1.12 and go modules 2019-06-12 13:12:00 +02:00
Andrea Maria Piana 4ab08629f6 Add postgres
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
2019-05-15 11:01:34 +02:00
Andrea Maria Piana 9e89efd859 Allow multiple db implementations
This commit creates an interface to use with the db so that we can
abstract what kind of db we use, therefore allowing us to chose db based
on config.
2019-05-15 11:01:34 +02:00
Andrea Maria Piana 061f10e58d
Publish rlp.RawValue instead of envelope ()
As part of a performance profiling of mailserver we noticed that most of
the resources on a query are spend decoding the whisper envelope.
This PR changes the way we store envelopes encoding the Topic into the
database key, so we can check that and we are able to publish the
envelope rawValue if it matches.
The change is backward compatible as only newly added envelopes will
have the new key, while old ones will have to be unmarshaled.
2019-05-09 12:58:02 +02:00
Andrea Maria Piana 354e6981ba
Add network incentivisation service () 2019-04-29 14:05:49 +02:00
Dmitry Shulyak 64188f0702
Deliver envelopes in forward order () 2019-04-23 10:04:58 +03:00
Andrea Maria Piana f5875d62c5
Fetch multiple nodes from contract () 2019-03-29 18:49:01 +01:00
Andrea Maria Piana 8baae97e2e Add GetNodesFromContract endpoint
We added an endpoint that given an RPC url and a contract will fetch a
list of nodes and return them to the client.
2019-03-29 12:55:54 +01:00
Andrea Maria Piana b574e3e53f
Default limit & upper bound ()
We allow the user not to specify an upper bound, in which case it will
default to Time.Now() + 60 seconds, to accomodate for messages with
higher timestamp.

Limit is also defaulted to 2000 if not set.

We removed as well the 24 hours limitation as now all the requests will
come with a limit.
2019-03-12 12:14:35 +01:00
Adam Babik 0c76505b95
expose more logs and metrics in DeliverMail () 2019-02-27 15:30:08 +01:00
Adam Babik 38aafe01db
Fix deadlock in mailserver DeliverMail method () 2019-02-22 09:55:37 +01:00
Adam Babik e38833b17a
add README for mailserver syncing () 2019-02-19 19:09:55 +01:00
Andrea Maria Piana da3964ba31
Change publishing envelopes behaviour () 2019-02-08 16:39:24 +01:00
Adam Babik d20b5dc3b3
mailserver: build pruning into WMailServer ()
MailServer pruning was implemented as a separate command but that required stopping a mail server and executing the command manually. This change builds pruning into MailServer and can be set using MailServerDataRetention in WhisperConfig.
2019-01-23 14:32:45 +01:00
Adam Babik 8f2e347e4f
mailserver: refactor mailserver's rate limiter () 2019-01-10 17:07:16 +01:00
Adam Babik 74cb16c456
add SyncMessages method to shhext api () 2018-12-14 12:21:34 +01:00
Adam Babik e2682486fd
Fix cursor encoding ()
It fixes encoding and decoding cursor sent in historic messages requests. `hex` package from the standard library is used.
2018-12-11 11:23:47 +01:00
Adam Babik 913dbfca9b
Support Mail Server data synchronization ()
These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
2018-12-06 10:48:28 +01:00
Adam Babik 860da591be
add mailserver batch requests counter and network processing time () 2018-11-19 10:14:03 +01:00
Adam Babik 52a1bdfed6
Upgrade geth 1.8.17 plus add metrics during compilation time ()
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.

The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
2018-11-14 08:03:58 +01:00
Adam Babik 0961e1039e
fix peerID logging in mail server () 2018-10-28 23:33:58 +01:00
Adam Babik a75f9c34cf
Request historical messages in batches () 2018-10-19 11:09:13 +02:00
Igor Mandrigin ba504e99c4
Mailserver: return error response. () 2018-10-18 12:25:00 +02:00
Andrea Franz b9baa48d87 Add metrics for mailserver errors () 2018-10-16 17:24:37 +02:00
Dmitry 5b551c67fa Use status-im whisper/whisperv6 module everywhere 2018-10-03 09:27:51 +03:00
Pedro Pombeiro 3d00af7fa3
Streamline configuration in status-go. Part of ()
- Replace command line flags with `-c` config flag. Part of 
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
2018-09-13 18:31:29 +02:00
Dmitry Shulyak f150d678de
Update geth to 1.8.14 ()
* Update to geth v1.8.14

* Remove patches that were merged upstream

* Apply patches before 0016

* Fix 0016 and apply it

* Apply everything else

* Pass gas limit as a second argument to simulated backend
2018-08-27 11:22:21 +03:00
Andrea Franz 874a3e8151
Feature/mailserver registry smart contract ()
* add verifier and test using simulated backend

* add ContractCaller

* commit simulated backend after deploy and after smart contract writes

* use bind.NewKeyedTransactor for all transactions in tests

* rename RegistryVerifier to Verifier

* initialize contract verifier if MailServerRegistryAddress config is set

* use contractAddress.Hash()

* refactoring

* use fmt.Sprintf to format contract address in logs

* fix test and lint warnings

* update Gopkg.lock

* update Gopkg.lock once more
2018-08-20 15:55:43 +02:00
Adrià Cidre 9ce96baf5d
[] Mailserver compatible with sync AND async keys () 2018-07-16 11:07:17 +02:00
Adam Babik 38a60135b2 Add asymmetric key support for MailServer requests ()
* add Asymmetric Key support for MailServer requests

* remove deprecated notice

* fix linter

* refactoring Whisper config related to MailServer

* fix race condition
2018-07-04 11:30:57 +02:00
Adrià Cidre 237aeea7b2
Centralize leveldb.OpenFile allowing Recover corrupted db everywhere () 2018-07-02 10:42:16 +02:00
Andrea Franz 8aa046e893
update mailserver tests to write to a temporary directory instead of using /tmp () 2018-07-02 09:44:02 +02:00
Andrea Franz 809db97e54
mailserver pagination ()
* mailserver sends envelopes in descending order

* add limit value in mailserver request payload

* mailserver sends messages up to the limit specified in the request

* update Archive method to return key and error

* processRequest returns the next page cursor

* add cursor to mailserver request

* add limit and cursor to request payload

* fix request limit encoding

* wait for request completed event in TrackerSuite/TestRequestCompleted

* add cursor to mailserver response

* fix cursor position in payload

* add e2e test for mail server pagination

* validate mail server response size

* remove old limitReached var

* fix lint warnings

* add whisper patch

* fix tests after rebase

* check all return values to avoid lint warnings

* check that all messages have been retrieved after 2 paginated requests

* fix lint warnings

* rename geth patch

* merge mailserver patches into one

* add last envelope hash to mailserver response and EventEnvelopeAvailable event

* update whisper patch

* add docs to MailServerResponse

* update whisper patch

* fix tests and lint warnings

* send mailserver response data on EventMailServerRequestCompleted signal

* update tracker tests

* optimise pagination test waiting for mailserver to archive only before requesting

* rollback mailserver interface changes

* refactoring and docs changes

* fix payload size check to determine if a limit is specified

* add more docs to the processRequest method

* add constants for request payload field lengths

* add const noLimits to specify that limit=0 means no limits
2018-07-02 09:38:10 +02:00
Andrea Franz 29d90b651d
test leveldb panics on mailserver ()
* test leveldb panics on mailserver

* refactoring

* rename DB interface to dbImpl and add documentation
2018-06-27 14:22:09 +02:00
Pedro Pombeiro ab7b9b914c Recover from panics which can get sent from goleveldb due to bad arguments. Closes 2018-06-26 16:49:13 +02:00
Ivan Daniluk b003400b4b Validate lower/upper limit in mailserver request () 2018-06-26 09:33:05 +02:00
Pedro Pombeiro fac5576988 Use `EnvelopeTracer` as an event source to avoid relying on timeouts in tests. Fixes
Fix bug in mailserver implementation which was using upper bound `to` parameter as an exclusive, rather than inclusive parameter, as stated in the documentation: https://github.com/status-im/status-go/wiki/Additional-JSON-RPC-API
2018-06-18 15:24:27 +02:00
Andrea Franz fa390a52ae
RequestHistoricMessages response ()
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Adam Babik b3a8991eb1
Update mailserver metrics for archived envelopes () 2018-06-15 11:19:17 +02:00
Pedro Pombeiro 354e23aaf5 Fix issues reported by lint. Part of 2018-06-14 13:52:51 +02:00
Adam Babik bbe7fba0c9
Benchmark tests ()
A set of tests used to benchmark Whisper and MailServer nodes.
2018-06-11 12:48:42 +02:00
Adrià Cidre d5be8c525d
[] move geth subpackages to root level () 2018-06-08 13:29:50 +02:00
Andrea Franz 4b938be856
add metrics to mailserver ()
* add metrics to mailserver

* sentEnvelopesSize existing sentEnvelopesSize
2018-05-30 14:31:47 +02:00
Adrià Cidre 4e33dfe7bc
Fix for mail server not limiting certain messages () 2018-05-22 17:51:05 +02:00
Adrià Cidre 06e64cdde2
Refactor mailserver tests () 2018-05-21 13:30:37 +02:00
Adrià Cidre e9da21cf87 Rollback mailserver DataDir to previous leveldb data directory 2018-05-20 08:34:47 +03:00
Andrea Franz 4317b8f687
add cmd/statusd-prune ()
add mailserver cleaner

use memstorage for leveldb in tests

avoid write if batch size is 0

add comments

add cmd/statusd-prune

rmeove batch size var in prune method

validate range values

pass only flag name to missingFlag

refactor Cleaner.prune method

update batch not to be a pointer

removed extra batch counter increment

don't increment counter if batch returns errors

add README
2018-05-17 19:24:00 +02:00
Adrià Cidre cf8ad7e002
[#ISSUE-947/ISSUE-946] Rate limit and batch limit on any mail server query () 2018-05-17 13:21:04 +02:00
Adrià Cidre 08b4d515c6
[#ISSUE-937] : Move mail server implementation to status-go repo ()
* [#ISSUE-937] : Move mail server implementation to status-go repo

* Applying dep-ensure
2018-05-11 21:43:07 +02:00