Commit Graph

82 Commits

Author SHA1 Message Date
Andrea Maria Piana 7387049d4b Upgrade linter and address issues 2020-12-28 16:55:14 +01:00
Samuel Hawksby-Robinson e8d883edb6 bindata for profile_image migrations 2020-12-17 14:10:00 +01:00
Andrea Maria Piana 0b2bd2863b Log mailserver error
while debugging I noticed we don't log `err` if the query fails.
2020-12-07 15:03:06 +01:00
Jonathan Rainville b2d8f7f416
feat: add bookmarks table for the browser bookmarks
plus get bookmark icon using a lib
2020-11-16 13:31:34 -05:00
Jakub Sokołowski 1e3801d105 use types.BytesToHash to print []byte variables
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-06-04 09:45:47 +02:00
Andrea Maria Piana aa7f591587 Move networking code for waku under `v0` namespace
Why make the change?

As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.

This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .

All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.

In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.

This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.

What has changed?

- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`

Things still to do

Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.

Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
2020-04-27 14:58:02 +02:00
Jakub Sokołowski c8d20f84f8 set archived_envelopes_total on start based on DB count
Before the count was reset every time process was restarted.
Also add 'db' label to distinguish between waku and whisper DBs.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-04-09 13:08:14 +02:00
Andrea Maria Piana 849492fda9
Add statusUpdate code (#1861)
* Add status-option code

This commits changes the behavior of waku introducing a new status-code,
`2`, that replaces the current single options codes.

* linting
2020-02-21 15:48:53 +01:00
Volodymyr Kozieiev 19fc448c85
pending-stickers field added to settings, unit-tests fixed (#1830) 2020-02-13 13:48:04 +02:00
Jakub Sokołowski cca47ef6e1 rename gomarkdown/markdown to status-im/markdown
This resolves a dependency conflict we have with MatterBridge
which was using a newer version of the same package.
This resulted in a JSON marshalling bug that would crash the bridge.

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-02-11 16:06:43 +01:00
Adam Babik bc2d018483
Add support for request messages by topics (#1805) 2020-01-21 08:11:24 +01:00
Adam Babik 87f1f8a965
Remove 24h time range validation from maislerver request (#1792) 2020-01-14 08:16:35 +01:00
Adam Babik 179e3e851e
Report correct number of pruned rows in Postgres mailserver (#1789) 2020-01-13 21:13:28 +01:00
Adam Babik 44aa313981
Make shhext and protocol work with Waku (#1777)
This change makes shhext and protocol submodule work with Waku and Whisper.
2020-01-13 20:17:30 +01:00
Andrea Maria Piana 88a1d0111e
Add Commands (#1731)
This commit adds handling of Transaction commands.
2020-01-10 19:59:01 +01:00
Adam Babik 37a508a97b
Integrate Waku service (#1759) 2020-01-08 12:12:23 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
yenda 024f30f0b9 Put settings in a table with schema (#1746)
Move settings table schema from a key-value store to a one row table with many columns.

We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.

Co-authored-by: Adam Babik <a.babik@designfortress.com>
2019-12-27 10:58:25 +01:00
André Medeiros 0571f561f0
APIs to Get and Create custom Tokens (#1717) 2019-12-10 12:31:08 -05:00
Adam Babik a636f33109
Set Whisper version submodule to v6 (#1725) 2019-12-09 11:36:14 +01:00
Adam Babik 4ac4a61e20
Move whisper to status-go monorepo (#1720) 2019-12-09 11:06:04 +01:00
Andrea Maria Piana fd49b0140e
Move to protobuf for Message type (#1706)
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`

* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.

* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.

* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
2019-12-05 17:25:34 +01:00
Pedro Pombeiro 26b6d7c36a Create a home submodule for Eth node bridges- Rename StatusBackend to GethStatusBackend 2019-11-27 17:02:09 +01:00
Pedro Pombeiro ed5a5c154d
Move to monorepo structure (#1684)
Move to a monorepo structure with submodules

- Rename status-protocol-go to status-go/protocol
2019-11-21 17:19:22 +01:00
Pedro Pombeiro 2dd74da23d Integrate Nimbus status-protocol-go 2019-11-18 14:30:03 +01:00
Corey 457acec72e
added name to batch loop so break works properly (#1650) 2019-11-06 09:12:05 -05:00
Adam Babik 183d626256
Implement mailserver Deliver() with MessagesRequest (#1653)
The main difference is that it takes `whisper.MessagesRequest` as an argument instead of `whisper.Envelope`.
2019-11-04 21:15:21 +01:00
Jakub Sokołowski c1f17cb4a3 use prometheus/client_golang for metrics
also re-implement mailserver metrics

Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-11-04 13:18:50 +01:00
Adam Babik c74c36742e
Log error in mailserver (#1644) 2019-10-15 16:33:58 +02:00
Andrea Maria Piana b27779aa4e
Add Mailserver Request Gaps service (#1596)
This commits add some endpoints to store mailserver request gaps,
and provides 4 endpoints:

```
AddMailserverRequestGaps(gaps []MailserverRequestGap) error
GetMailserverRequestGaps(chatID string) []MailserverRequestGap
DeleteMailserverRequestGaps(ids []string) error
DeleteMailserverRequestGapsByChatID(id string) error
```
2019-09-04 12:04:17 +02:00
Adam Babik d7f454fe4c
Add mailservers service (#1595) 2019-09-04 08:25:33 +02:00
Adam Babik 1ab2e88bf5
Sync messages in a loop until error or no cursor (#1502) 2019-06-26 18:17:41 +02:00
Andrea Maria Piana cef7f367ab Add topic negotiation
This commit add topic negotiation to the protocol.

On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.

This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).

When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.

This change is backward compatible, as long as there is no downgrade of
the app on the other side.

A few changes:

* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).

* Factored out multidevice management in a separate namespace

* Moved all the test to test the whole protoocl rather than just the encryption service

* Moved all the filter management in status-go
2019-06-19 17:38:45 +02:00
Adam Babik c9e99c432d migrate to go 1.12 and go modules 2019-06-12 13:12:00 +02:00
Andrea Maria Piana 4ab08629f6 Add postgres
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
2019-05-15 11:01:34 +02:00
Andrea Maria Piana 9e89efd859 Allow multiple db implementations
This commit creates an interface to use with the db so that we can
abstract what kind of db we use, therefore allowing us to chose db based
on config.
2019-05-15 11:01:34 +02:00
Andrea Maria Piana 061f10e58d
Publish rlp.RawValue instead of envelope (#1459)
As part of a performance profiling of mailserver we noticed that most of
the resources on a query are spend decoding the whisper envelope.
This PR changes the way we store envelopes encoding the Topic into the
database key, so we can check that and we are able to publish the
envelope rawValue if it matches.
The change is backward compatible as only newly added envelopes will
have the new key, while old ones will have to be unmarshaled.
2019-05-09 12:58:02 +02:00
Andrea Maria Piana 354e6981ba
Add network incentivisation service (#1452) 2019-04-29 14:05:49 +02:00
Dmitry Shulyak 64188f0702
Deliver envelopes in forward order (#1437) 2019-04-23 10:04:58 +03:00
Andrea Maria Piana f5875d62c5
Fetch multiple nodes from contract (#1430) 2019-03-29 18:49:01 +01:00
Andrea Maria Piana 8baae97e2e Add GetNodesFromContract endpoint
We added an endpoint that given an RPC url and a contract will fetch a
list of nodes and return them to the client.
2019-03-29 12:55:54 +01:00
Andrea Maria Piana b574e3e53f
Default limit & upper bound (#1411)
We allow the user not to specify an upper bound, in which case it will
default to Time.Now() + 60 seconds, to accomodate for messages with
higher timestamp.

Limit is also defaulted to 2000 if not set.

We removed as well the 24 hours limitation as now all the requests will
come with a limit.
2019-03-12 12:14:35 +01:00
Adam Babik 0c76505b95
expose more logs and metrics in DeliverMail (#1398) 2019-02-27 15:30:08 +01:00
Adam Babik 38aafe01db
Fix deadlock in mailserver DeliverMail method (#1388) 2019-02-22 09:55:37 +01:00
Adam Babik e38833b17a
add README for mailserver syncing (#1377) 2019-02-19 19:09:55 +01:00
Andrea Maria Piana da3964ba31
Change publishing envelopes behaviour (#1371) 2019-02-08 16:39:24 +01:00
Adam Babik d20b5dc3b3
mailserver: build pruning into WMailServer (#1342)
MailServer pruning was implemented as a separate command but that required stopping a mail server and executing the command manually. This change builds pruning into MailServer and can be set using MailServerDataRetention in WhisperConfig.
2019-01-23 14:32:45 +01:00
Adam Babik 8f2e347e4f
mailserver: refactor mailserver's rate limiter (#1341) 2019-01-10 17:07:16 +01:00
Adam Babik 74cb16c456
add SyncMessages method to shhext api (#1309) 2018-12-14 12:21:34 +01:00
Adam Babik e2682486fd
Fix cursor encoding (#1308)
It fixes encoding and decoding cursor sent in historic messages requests. `hex` package from the standard library is used.
2018-12-11 11:23:47 +01:00