Commit Graph

115 Commits

Author SHA1 Message Date
Richard Ramos 0575030825 feat: adding peers, fix peer count and fix format of peer stats signal 2021-09-27 09:04:28 -04:00
Roman Volosovskyi 839d6b3194
[status-im/status-react#12517] Ensure that all necessary contact fields are synced 2021-09-23 12:22:49 +03:00
Richard Ramos a8a5acabb2 fix: sqlite variable limit 2021-09-22 08:04:28 -04:00
Samuel Hawksby-Robinson 07e46714f0
Anon Metrics Broadcast (#2198)
* Protobufs and adapters

* Added basic anon metric service and config init

* Added fibonacci interval incrementer

* Added basic Client.Start func and integrated interval incrementer

* Added new processed field to app metrics table

* Added id column to app metrics table

* Added migration clean up

* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests

There was a wierd bug where metrics in the db that did not explicitly insert a  value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.

* Updated adaptors and db to handle proto_id

I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.

* Moved incrementer into dedicated file

* Resolve incrementer test fail

* Finalised the main loop functionality

* Implemented delete loop framework

* Updated adaptors file name

* Added delete loop delay and quit, and tweak on RawMessage gen

* Completed delete loop logic

* Added DBLock to prevent deletion during mainLoop

* Added postgres DB connection, integrated into anonmetrics.Server

* Removed proto_id from SQL migration and model

* Integrated postgres with Server and updated adaptors

* Function name update

* Added sample config files for client and server

* Fixes and testing for low level e2e

* make generate

* Fix lint

* Fix for receiving an anonMetricBatch not in server mode

* Postgres test fixes

* Tidy up, make vendor and make generate

* delinting

* Fixing database tests

* Attempted fix of does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error on sql resource loas

* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work

* Lint for the lint god

* Why doesn't the linter list all its problems at once?

* test tweaks

* Fix for wakuV2 change

* DB reset change

* Fix for postgres db migrations fails

* More robust implementation of postgres test setup and teardown

* Added block for anon metrics functionality

* Version Bump to 0.84.0

* Added test to check anon metrics broadcast is deactivated

* Protobufs and adapters

* Added basic anon metric service and config init

* Added new processed field to app metrics table

* Added id column to app metrics table

* Added migration clean up

* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests

There was a wierd bug where metrics in the db that did not explicitly insert a  value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.

* Updated adaptors and db to handle proto_id

I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.

* Added postgres DB connection, integrated into anonmetrics.Server

* Removed proto_id from SQL migration and model

* Integrated postgres with Server and updated adaptors

* Added sample config files for client and server

* Fix lint

* Fix for receiving an anonMetricBatch not in server mode

* Postgres test fixes

* Tidy up, make vendor and make generate

* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
2021-09-01 13:02:18 +01:00
RichΛrd 842ba229df
feat: add function to obtain bloomfilter (#2334) 2021-08-26 16:25:43 -04:00
Samuel Hawksby-Robinson 89251e8416
Sync Communities (#2253)
* Added community sync protobuf

* Updated community sync send logic

* Integrated syncCommunity handling

* Added synced_at field and tidied up some other logic

* persistence testing

* Added testing and join functionality

* Fixed issue with empty scan params

* Finshed persistence tests for new db funcs

* Midway debug of description not persisting after sync

* Resolved final issues and tidied up

* Polish

* delint

* Fix error not handled on SetPrivateKey

* fix infinite loop, again

* Added muted option and test fix

* Added Muted to syncing functions, not just in persistence

* Fix bug introduced with Muted property

* Added a couple of notes for future devs

* Added most of the sync RequestToJoin functionality

Tests need to be completed and tests are giving some errors

* Finished tests for getJoinedAndPending

* Added note

* Resolving lint

* Fix of protobuf gen bug

* Fixes to community sync tests

* Fixes to test

* Continued fix of e2e

* Final fix to e2e testing

* Updated migration position

* resolve missing import

* Apparently the linter spellchecks

* Fix bug from  #2276 merge

* Bug fix for leaving quirkiness

* Addressed superfluous MessengerResponse field

* Addressed feedback

* VERSION bump
2021-08-06 16:40:23 +01:00
RichΛrd facad9f07e
feat: upload/download rate for waku v1 messages (#2286)
* feat: upload/download rate for waku v1 messages

* reorganize code

* fix failing test
2021-08-03 15:27:15 -04:00
Jonathan Rainville cdc7c55030
Delete messages (#2279)
* feat: add api and handling to delete messages

* fix(delete): add whitelist to message types that can be deleted
2021-07-26 17:06:32 -04:00
RichΛrd bfdc000bbc
go-waku: RequestAllHistoricMessages (#2258)
* feat: update wakuv2 store

* check online using waku version for now

* fix peerId and from/to multiplier

* fix: lint

* fix: handle waku2 cursors

* fix: code review

* fix: code review 2

* update go-waku version

* update vendor folder
2021-07-21 15:02:50 -04:00
Andrea Maria Piana f31dc5df92 Use marshal binary 2021-07-20 10:57:38 +02:00
Andrea Maria Piana 799bd93451 Memove mailserver registry and update ierc20 contract bindings
This commit removes the mailserver contract registry, as not used
anymore, and upgrade the ERC20 contract bindings.
2021-07-20 10:57:38 +02:00
Brian Sztamfater 92404a5ab7
Add reply message to activity center notification of type reply (#2272) 2021-07-15 17:21:44 -03:00
Jonathan Rainville 11c46edd8b
feat(community): add muted to community and function to set it (#2271) 2021-06-30 09:29:43 -04:00
Richard Ramos 217bace570 feat: edit messages 2021-06-29 13:15:15 +02:00
Jonathan Rainville 14cfcebb5a
feat: add author to notification in activity center (#2259) 2021-06-17 11:08:28 -04:00
RichΛrd 40359f9c1b
go-waku integration (#2247)
* Adding wakunode module
* Adding wakuv2 fleet files
* Add waku fleets to update-fleet-config script
* Adding config items for waku v2
* Conditionally start waku v2 node depending on config
* Adapting common code to use go-waku
* Setting log level to info
* update dependencies
* update fleet config to use WakuNodes instead of BootNodes
* send and receive messages
* use hash returned when publishing a message
* add waku store protocol
* trigger signal after receiving store messages
* exclude linting rule SA1019 to check deprecated packages
2021-06-16 16:19:45 -04:00
Andrea Maria Piana af1e47c258 Fix group chat filters 2021-06-01 12:59:52 +02:00
Andrea Maria Piana 3a61c3bae9 Allow setting mailserver sync value
This commit adds a setting for mailserver, that allows the client to
specify the syncing period.
Also fixes a minor issue with deletion of public chats.
2021-06-01 12:59:52 +02:00
Brian Sztamfater a72c32009a
Add mention notification to activity center (#2239)
Signed-off-by: Brian Sztamfater <brian@status.im>
2021-05-29 14:05:25 -03:00
Andrea Maria Piana dfd4668087 Add mentioned field
This commit adds a Mentioned field to Message, which indicates whether
the receiving user is mentioned in the message.
2021-05-26 08:33:38 +02:00
Andrea Maria Piana d50fee6bb2 Handle connection state 2021-05-21 07:22:58 +02:00
Andrea Maria Piana be01875d1d remove mailserver logic 2021-05-21 07:22:58 +02:00
Andrea Maria Piana cf6ef3171a Add mailserver logic 2021-05-21 07:22:58 +02:00
Pascal Precht dd49e604d4 feat: introduce new `joined` property in `Chat` struct
This property is useful for clients to know when a channel or chat
was joined so they can use that to calculate the order of channels
and chats shown in applications.

Changes include a new joined property on the Chat struct,
as well as adjustments in the persistence layer to retreive and
update chat data in the database.

In addition there's a migration script that alters the existing
chat table to introduce a new column for the joined field.
It also updates all existing rows in the database to set `joined`
to `0`.
2021-05-18 11:29:03 +02:00
Roman Volosovskyi 6037570901
[wallet] Pending transactions API 2021-04-23 18:20:38 +03:00
Volodymyr Kozieiev 9241903edb
Refactorings: whisper transport removed (#2200) 2021-04-22 16:36:18 +03:00
Volodymyr Kozieiev 8a370c1c41
Request info about community from mailserver (#2177) 2021-04-19 15:09:46 +03:00
Andrea Maria Piana c739f73f49 Add activity center & messages from contacts only 2021-04-16 20:42:40 +02:00
Samuel Hawksby-Robinson 6d7028a8d9
Added Anonymous Metrics Send Opt-In Setting (#2195)
* Added anon metrics send opt in setting

* resolved rebase conflict, renamed migration to use unixtimestamp

Theres always conflicts with migrations using sequential numbers, less so with unix timestamp
2021-04-13 14:00:18 +01:00
Shivek Khurana 79bf90e990
👮🏽‍♂️ Update validators and add a new event (#2189)
* Add extra event to capture other type of navigations, allow empty screen name, rename cofx to get rid of clj ns, update tests

* Some view ids are greater than 16 characters. Made it 32 to be safe.

* Tab navigation events occur outside nav, add a new validator for them

* Remove navigate to cofx event, capture screens on will focus, get rid of enum and make valid screens a string less than 32 characters

* Run make generate

* Fix test

* Bump version to 0.75.1
2021-04-12 17:55:53 +05:30
Andrea Maria Piana d22182ea82 Ban user from community
This commit introduces a new endpoint to ban a user from a community.

Messages from this user will be ignored.
2021-03-24 09:29:21 +01:00
Shivek Khurana 1097b14a7f
🎭 📊 Anonymous Metrics V0 (#2170)
* Migrations in place, how to run them?

* Remove down migrations and touch database.go

* Database and Database Test package in place, added functions to get and store app metrics

* make generate output

* Minor bug fix on app metrics insert and select

* Add a validation layer to restrict what can be saved in the database

* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service

* service updates

* Compute all errors before sending them out

* Trying to bring a closjure to appmetrics go

* Expose appmetrics via an api, skip fancy

* Address value as Jason Dawt Rawmasage to ease parsing

* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.

* Lint issues

* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel

* Bump migration number

* Fix API factory usage

* Add comment re:json.RawMessage instead of strings

* Get rid of test vars, throw save error inside the loop

* Update version

Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2021-03-17 18:09:28 +05:30
Andrea Maria Piana f75f7bb738 Expand confirmations to private group chat messages
This commit expands the confirmation mechanism to allow private group
chat messages to be confirmed:

Changes:

- Added a separate table for message confirmations as group chat
messages have same messageID but multiple datasyncID
- Removed DataSyncID from raw message (I haven't removed the column name
as it can't be done in sqlite without copying over the table)
2021-03-16 13:41:14 +01:00
Roman Volosovskyi a7f2e94999
[wallet] Store account's last known balance and nonce 2021-03-15 16:18:12 +02:00
Andrea Maria Piana f115b8d289 Request/Decline access to communities 2021-02-26 15:35:43 +01:00
Andrea Maria Piana a1b3e3a772
Clean topics that we don't listen to
There was a bug on status-react where it would save filters that were
not listened to.
This commit adds a task to clean up those filters as they might result
in long syncing times.

This commit also returns topics/ranges/mailserves from messenger in
order to make the initialization of the app simpler and start moving
logic to status-go.

It also removes whisper from vendor.
2021-01-26 09:39:57 +01:00
Andrea Maria Piana fb6411af24
Use personal topic for push notification registration
One of the issues we noticed is that the partitioned topic
in push notification is heavy in traffic, as any user using a particular
mailserver will use that partitioned topic to register for PNs.

This commit moves from the partitioned topic to the personal topic of
the PN server, so it does not clash with other users that might happen
to have the same partitioned topic as the mailserver, resulting in long
sync times.

Another issue that will need to be addressed separately is that once you
send a message to a topic, because of the way how waku/whisper works,
you will have to register to that topic, meaning that you will receive
that data. Currently waku does not support unsubscribing from a topic
without logging in and out, so that needs also to be addressed.
2021-01-26 09:39:53 +01:00
Andrea Maria Piana e3969a7752 Cache waku messages
This commit re-introduces a feature that we lost during the migration to
status-go.
Messages are cached for a couple of days if processed correctly by
status-go, to avoid performance issues.
2021-01-18 09:38:27 +01:00
Andrea Maria Piana b5b1e19c24 Add ProfilePicturesVisibility 2021-01-15 13:39:13 +01:00
Andrea Maria Piana f5482ec187 Remove filters 2020-12-23 17:20:55 +01:00
Andrea Maria Piana 5d929c3584 Handle organisation membership requests 2020-12-23 17:20:55 +01:00
Samuel Hawksby-Robinson e8d883edb6 bindata for profile_image migrations 2020-12-17 14:10:00 +01:00
Volodymyr Kozieiev d4518d7f16
resend emoji reaction (#2088) 2020-12-15 16:43:41 +02:00
Jonathan Rainville b2d8f7f416
feat: add bookmarks table for the browser bookmarks
plus get bookmark icon using a lib
2020-11-16 13:31:34 -05:00
Andrea Maria Piana 75e0809f50 Split datasync messages in batches
When sending messages in quick succession, it might be that multiple
messages are batched together in datasync, resulting in a single large
payload.
This commit changes the behavior so that we can pass a max-message-size
and we split the message in batches before sending.

A more elegant way would be to split at the transport layer (i.e
waku/whisper), but that would be incompatible with older client.

We can still do that eventually to support larger messages.
2020-11-05 12:45:29 +01:00
Andrea Maria Piana 63af6aa79b
Use ephemeral keys for sending messsages 2020-07-27 08:51:27 +02:00
Andrea Maria Piana 2d17c40631 Refactor *Raw methods in message_processor.go
We used *Raw method in message_processor as before we had non-Raw method
during the transition from status-react to status-go. This naming is not
meaningful anymore, so I have changed it.
2020-06-01 11:30:04 +02:00
Andrea Maria Piana 8ba6625df0 Lower PoW for messages
Images are too large to be sent over waku/whisper with the
current PoW (0.002).

This commit lowers the PoW for sending messages to a lower value,
depending on the size.

This means that older clients will not be able to receive messages that
are greater than 50KB (none of the messages we currently send is anywhere close to
that, most of them are less than 1KB).

I would have preferred to set the PoW explicitly for images to be lower,
but it's not trivial as we use `datasync` to send messages and its
interface takes a payload that is to be dispatched and a function to
dispatch, at initialization stage.

This make it difficult (impossible?) to set a different PoW for a
particular message, without changing function signature in datasync
(which is agnostic to the transport used).

So a less cumbersome approach is to just fingerprint on size.
2020-06-01 11:30:04 +02:00
Andrea Maria Piana f5ab58b87f Add support for images
This commit adds support for images in protobuf messages.

The client can specify a path which will be used to load the image
and set the corresponding fields.

This makes the assumption that the RCP server runs on the same machine
as the client and they have access to the same files. This holds
currently for both status-react and status-console-client, we could
revisit and adds an upload if that changes in the future.
2020-06-01 11:30:04 +02:00
André Medeiros be683556ff
fix: close resultsets so we don't leak them (#1809)
* fix: close resultsets so we don't leak them

* Refactor browsers/database

To implement PR suggestions and improve code quality.

* Refactor services/permissions/database

To implement PR suggestions and improve code quality.

Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
2020-05-14 11:51:32 +01:00
Adam Babik 831ab85561
Upgrade deps (#1892) 2020-03-09 08:55:58 +01:00
Adam Babik 8b61d92d58
Remove protocol and eth-node submodules (#1835) 2020-02-10 12:22:37 +01:00
Andrea Maria Piana f2eebd1e49
Verify ENS in the background (#1824)
Currently ENS are verified explicitly by status-react, this is not ideal
as if that fails it will have to be explicilty retried in status-react.
This commits changes that behavior so that ENS are verified in a loop
and updated if new messages are received.
2020-02-05 11:09:33 +01:00
Pedro Pombeiro 741f43cb0c Fix wrong field being logged in `EnvelopesMonitor` 2020-01-23 17:19:10 +01:00
Andrea Maria Piana 456bcfa022
Peg clock value to whisper timestamp (#1804)
This commit pegs the clock value to maximum + 120 seconds from the whisper
timestamp.
In this way the we avoid the scenario where a client makes the timestamp
increase arbitrarely.
2020-01-20 17:44:32 +01:00
Andrea Maria Piana 25d46c6d82
Fix waku tests & contact ens (#1802) 2020-01-17 13:39:09 +01:00
Pedro Pombeiro 7f45c43196 make lint-fix 2020-01-16 13:18:36 +01:00
Adam Babik 44aa313981
Make shhext and protocol work with Waku (#1777)
This change makes shhext and protocol submodule work with Waku and Whisper.
2020-01-13 20:17:30 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
Pedro Pombeiro 0cf24d2c3e Remove unnecessary call to `GetFilter` 2019-12-19 14:11:48 +01:00
Adam Babik a636f33109
Set Whisper version submodule to v6 (#1725) 2019-12-09 11:36:14 +01:00
Adam Babik 4ac4a61e20
Move whisper to status-go monorepo (#1720) 2019-12-09 11:06:04 +01:00
Andrea Maria Piana fd49b0140e
Move to protobuf for Message type (#1706)
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`

* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.

* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.

* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
2019-12-05 17:25:34 +01:00
Pedro Pombeiro 26b6d7c36a Create a home submodule for Eth node bridges- Rename StatusBackend to GethStatusBackend 2019-11-27 17:02:09 +01:00
Pedro Pombeiro ed5a5c154d
Move to monorepo structure (#1684)
Move to a monorepo structure with submodules

- Rename status-protocol-go to status-go/protocol
2019-11-21 17:19:22 +01:00