These APIs are being introduced to address #2706 and #2704, provided
that clients will move to using these APIs instead of the currently
provided equivalent APIs in the browser service.
The `bookmarks` table is being extended with a `deleted_at` field which
can later be used for garbage collection, as "removing" a bookmark is
merely soft deletion and doesn't actually remove the data from the
database.
In addition to those APIs adding and soft deleting bookmark entries,
they also automatically perform a sync operation to ensure that
bookmarks are synced in real-time (#2704).
Closes#2706, #2704
This commit introduces a new `clock` field in the
`communities_settings` table so that it can be leveraged for syncing
community settings across devices.
It alsoe exends existing `syncCommunity` APIs to generate
`SyncCommunitySettings` as well, avoiding sending additional sync messages
for community settings.
When editing communities however, we still sync community settings
explicitly are we aren't syncing the community itself in that case.
This allows to store community admin settings that are meant to be propagated
to community members (as opposed to the already existing
`CommunitySettings` which are considered local to every account).
The first setting introduced as part of this commit is one that enables
community admins to configure whether or not members of the community
are allowed to pin messages in community channels.
Prior to this commit, this was not restricted at all on the protocol
level and only enforced by clients via UI (e.g. members don't see an
option to pin messages, although they could).
This config setting now ensures that:
1. If turned off, members cannot send a pin message
2. If turned off, pin messages from members are not handled/processed
This is needed by https://github.com/status-im/status-desktop/issues/5662
This introduces the ability for status notes to handle community
history archive magnetlinks. To make this work, a few things are needed:
1. A new database table has been introduced to store message archive
hashes. This is necessary so status nodes can determine whether or
not they need to download a certain archive
2. The messenger's `handleRetrievedMessages()` has been exteded to take
magnetlink messages into account
3. New APIs were added to download torrent data given a magnetlink and
also to extract messages from downloaded archives, which are then
later fed to `handleRetrievedMessages`
Closes#2568
This introduces logic needed to:
- Create WakuMessageArchives and and indices from store waku messages
- History archive torrent data to disk and create .torrent file from
that
- Seed and unseed history archive torrents as necessary
- Starting/stopping the torrent client
- Enabling/disabling community history support for individual components
and starting/stopping the routine intervals accordingly
This does not yet handle magnet links (#2568)
Closes#2567
This is needed so that when they are bundled into archives, receiving
nodes can still verify the messages payload using its signature.
This commit introduces a new `waku_messages` table and APIs to store
such messages. Waku message payload is store for any message that has
a topic that matches any of the admin communities chats.
Closes#2566
* Sync Settings
* Added valueHandlers and Database singleton
Some issues remain, need a way to comparing incoming sql.DB to check if the connection is to a different file or not. Maybe make singleton instance per filename
* Added functionality to check the sqlite filename
* Refactor of Database.SaveSyncSettings to be used as a handler
* Implemented inteface for setting sync protobuf factories
* Refactored and completed adhoc send setting sync
* Tidying up
* Immutability refactor
* Refactor settings into dedicated package
* Breakout structs
* Tidy up
* Refactor of bulk settings sync
* Bug fixes
* Addressing feedback
* Fix code dropped during rebase
* Fix for db closed
* Fix for node config related crashes
* Provisional fix for type assertion - issue 2
* Adding robust type assertion checks
* Partial fix for null literal db storage and json encoding
* Fix for passively handling nil sql.DB, and checking if elem has len and if len is 0
* Added test for preferred name behaviour
* Adding saved sync settings to MessengerResponse
* Completed granular initial sync and clock from network on save
* add Settings to isEmpty
* Refactor of protobufs, partially done
* Added syncSetting receiver handling, some bug fixes
* Fix for sticker packs
* Implement inactive flag on sync protobuf factory
* Refactor of types and structs
* Added SettingField.CanSync functionality
* Addressing rebase artifact
* Refactor of Setting SELECT queries
* Refactor of string return queries
* VERSION bump and migration index bump
* Deactiveate Sync Settings
* Deactiveated preferred_name and send_status_updates
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit introduces a new `TorrentConfig` as per #2563 with dedicated
default values and new options/flags for the status-go CLI.
Since it's part of `NodeConfig`, which is stored per user in the
database, this commit also adds a migration script that adds a new
`torrent_config` table and database APIs to insert and retreive the
torrent config.
Closes#2563
This commit enables mailserver cycle logic by default and make a few
changes:
1) Nodes are graylisted instead of being blacklisted for a set amount of
time. The reason is that if we blacklist, any cut in connectivity
might result in long delays before reconnecting, especially on spotty
connections.
2) Fixes an issue on the devp2p server, whereby the node would not
connect to one of the static nodes since all the connection slots
where filled. The fix is a bit inelegant, it always connects to
static nodes, ignoring maxpeers, but it's tricky to get it to work
since the code is clearly not written to select a specific node.
3) Adds support to pinned mailservers
4) Add retries to mailservers requests. It uses a closure for now, I
think we should eventually have a channel etc, but I'd leave that for
later.
* Protobufs and adapters
* Added basic anon metric service and config init
* Added fibonacci interval incrementer
* Added basic Client.Start func and integrated interval incrementer
* Added new processed field to app metrics table
* Added id column to app metrics table
* Added migration clean up
* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests
There was a wierd bug where metrics in the db that did not explicitly insert a value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.
* Updated adaptors and db to handle proto_id
I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.
* Moved incrementer into dedicated file
* Resolve incrementer test fail
* Finalised the main loop functionality
* Implemented delete loop framework
* Updated adaptors file name
* Added delete loop delay and quit, and tweak on RawMessage gen
* Completed delete loop logic
* Added DBLock to prevent deletion during mainLoop
* Added postgres DB connection, integrated into anonmetrics.Server
* Removed proto_id from SQL migration and model
* Integrated postgres with Server and updated adaptors
* Function name update
* Added sample config files for client and server
* Fixes and testing for low level e2e
* make generate
* Fix lint
* Fix for receiving an anonMetricBatch not in server mode
* Postgres test fixes
* Tidy up, make vendor and make generate
* delinting
* Fixing database tests
* Attempted fix of does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error on sql resource loas
* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
* Lint for the lint god
* Why doesn't the linter list all its problems at once?
* test tweaks
* Fix for wakuV2 change
* DB reset change
* Fix for postgres db migrations fails
* More robust implementation of postgres test setup and teardown
* Added block for anon metrics functionality
* Version Bump to 0.84.0
* Added test to check anon metrics broadcast is deactivated
* Protobufs and adapters
* Added basic anon metric service and config init
* Added new processed field to app metrics table
* Added id column to app metrics table
* Added migration clean up
* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests
There was a wierd bug where metrics in the db that did not explicitly insert a value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.
* Updated adaptors and db to handle proto_id
I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.
* Added postgres DB connection, integrated into anonmetrics.Server
* Removed proto_id from SQL migration and model
* Integrated postgres with Server and updated adaptors
* Added sample config files for client and server
* Fix lint
* Fix for receiving an anonMetricBatch not in server mode
* Postgres test fixes
* Tidy up, make vendor and make generate
* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does: cannot open `does' (No such file or directory)
not: cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
* Added community sync protobuf
* Updated community sync send logic
* Integrated syncCommunity handling
* Added synced_at field and tidied up some other logic
* persistence testing
* Added testing and join functionality
* Fixed issue with empty scan params
* Finshed persistence tests for new db funcs
* Midway debug of description not persisting after sync
* Resolved final issues and tidied up
* Polish
* delint
* Fix error not handled on SetPrivateKey
* fix infinite loop, again
* Added muted option and test fix
* Added Muted to syncing functions, not just in persistence
* Fix bug introduced with Muted property
* Added a couple of notes for future devs
* Added most of the sync RequestToJoin functionality
Tests need to be completed and tests are giving some errors
* Finished tests for getJoinedAndPending
* Added note
* Resolving lint
* Fix of protobuf gen bug
* Fixes to community sync tests
* Fixes to test
* Continued fix of e2e
* Final fix to e2e testing
* Updated migration position
* resolve missing import
* Apparently the linter spellchecks
* Fix bug from #2276 merge
* Bug fix for leaving quirkiness
* Addressed superfluous MessengerResponse field
* Addressed feedback
* VERSION bump
* Adding wakunode module
* Adding wakuv2 fleet files
* Add waku fleets to update-fleet-config script
* Adding config items for waku v2
* Conditionally start waku v2 node depending on config
* Adapting common code to use go-waku
* Setting log level to info
* update dependencies
* update fleet config to use WakuNodes instead of BootNodes
* send and receive messages
* use hash returned when publishing a message
* add waku store protocol
* trigger signal after receiving store messages
* exclude linting rule SA1019 to check deprecated packages
This commit adds a setting for mailserver, that allows the client to
specify the syncing period.
Also fixes a minor issue with deletion of public chats.
This property is useful for clients to know when a channel or chat
was joined so they can use that to calculate the order of channels
and chats shown in applications.
Changes include a new joined property on the Chat struct,
as well as adjustments in the persistence layer to retreive and
update chat data in the database.
In addition there's a migration script that alters the existing
chat table to introduce a new column for the joined field.
It also updates all existing rows in the database to set `joined`
to `0`.
* Added anon metrics send opt in setting
* resolved rebase conflict, renamed migration to use unixtimestamp
Theres always conflicts with migrations using sequential numbers, less so with unix timestamp
* Add extra event to capture other type of navigations, allow empty screen name, rename cofx to get rid of clj ns, update tests
* Some view ids are greater than 16 characters. Made it 32 to be safe.
* Tab navigation events occur outside nav, add a new validator for them
* Remove navigate to cofx event, capture screens on will focus, get rid of enum and make valid screens a string less than 32 characters
* Run make generate
* Fix test
* Bump version to 0.75.1
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
This commit expands the confirmation mechanism to allow private group
chat messages to be confirmed:
Changes:
- Added a separate table for message confirmations as group chat
messages have same messageID but multiple datasyncID
- Removed DataSyncID from raw message (I haven't removed the column name
as it can't be done in sqlite without copying over the table)
There was a bug on status-react where it would save filters that were
not listened to.
This commit adds a task to clean up those filters as they might result
in long syncing times.
This commit also returns topics/ranges/mailserves from messenger in
order to make the initialization of the app simpler and start moving
logic to status-go.
It also removes whisper from vendor.
In some instances the communities migration would be skipped but not
marked as `dirty`.
This commit addresses the issue by:
- Making sure that if dirty is set the migration is not skipped but
replayed
- If the version is on the communities migration and dirty is false, we
check for the presence of the communities table. If not present we
replay the communities migration.
- Make community_id field in user_messages nullable
It also removes all the `down` migration, as we can't use them
effectively, as explained in the README.md added.
Currently replies to messages are handled in status-react.
This causes some issues with the fact that sometimes replies might come
out of order, they might be offloaded to the database etc.
This commit changes the behavior so that status-go always returns the
replies, and in case a reply comes out of order (first the reply, later
the message being replied to), it will include in the messages the
updated message.
It also adds some fields (RTL,Replace,LineCount) to the database which
were not previously saved, resulting in some potential bugs.
The method that we use to pull replies is currently a bit naive, we just
pull all the message again from the database, but has the advantage of
being simple. It will go through performance testing to make sure
performnace are acceptable, if so I think it's reasonable to avoid some
complexity.
* Add status-option code
This commits changes the behavior of waku introducing a new status-code,
`2`, that replaces the current single options codes.
* linting
Move settings table schema from a key-value store to a one row table with many columns.
We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.
Co-authored-by: Adam Babik <a.babik@designfortress.com>
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`
* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.
* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.
* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
This change flattens messaging/chat package. It also removes dependency between multidevice and chat/protobuf packages.
The Publisher interface was also changed a bit to support more native types.
Version got bumped to 0.29.0-beta.3.
* Move installations to status-go
This commit moves installations management/storage to status-go.
We remove the native binding and provide RPC endpoints to set the
metadata and return a list of our own installations.
* Cache keys
Generating a symkey can take up to a second on slow devices, this commit
makes so that keys are saved once generated and stored in the database.
This commit add topic negotiation to the protocol.
On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.
This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).
When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.
This change is backward compatible, as long as there is no downgrade of
the app on the other side.
A few changes:
* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).
* Factored out multidevice management in a separate namespace
* Moved all the test to test the whole protoocl rather than just the encryption service
* Moved all the filter management in status-go
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
Change to support sending multiple bundles, as needed for group chats,
limit number of devices to 3 as already done in the UI and refresh
bundle daily.
This commit adds a list new table, installations, which is used to keep
track of which installation are active for a given identity key.
In general, we limit the number of installation that we keep
synchronized to 5, to avoid excessive usage of resources.
Any installation coming from our own identity, will have to be manually
enabled, otherwise we trust the other peer has correctly paired their
devices.
We use a timestamp to decide which installations to keep synchronized as
a logical clock would have make the creation of the bundle more
complicated, but this can always be converted to a logical clock at
later stages without breaking compatibility.
- Skipped keys
The purpose of limiting the number of skipped keys generated is to avoid a dos
attack whereby an attacker would send a large N, forcing the device to
compute all the keys between currentN..N .
Previously the logic for handling skipped keys was:
- If in the current receiving chain there are more than maxSkip keys,
throw an error
This is problematic as in long-lived session dropped/unreceived messages starts
piling up, eventually reaching the threshold (1000 dropped/unreceived
messages).
This logic has been changed to be more inline with signals spec, and now
it is:
- If N is > currentN + maxSkip, throw an error
The purpose of limiting the number of skipped keys stored is to avoid a dos
attack whereby an attacker would force us to store a large number of
keys, filling up our storage.
Previously the logic for handling old keys was:
- Once you have maxKeep ratchet steps, delete any key from
currentRatchet - maxKeep.
This, in combination with the maxSkip implementation, capped the number of stored keys to
maxSkip * maxKeep.
The logic has been changed to:
- Keep a maximum of MaxMessageKeysPerSession
and additionally we delete any key that has a sequence number <
currentSeqNum - maxKeep
- Version
We check now the version of the bundle so that when we get a bundle from
the same installationID with a higher version, we mark the previous
bundle as expired and use the new bundle the next time a message is sent
- Replace command line flags with `-c` config flag. Part of #1180
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
In the current codebase if you want to use different keys (or any
other configuration) for tests it is required to place them at a
certain path in static folder, bundle them with go-bindata and only
then run tests. This is simply unnecessary.
This commit instead directs loads configurations from file without
relying on go-bindata
* Rename bootnode to static peers
Also add some groupings between types.
* Remove not needed genesisHash
* More cleanup of bootnodes with static peers
* Add option of cluster configuration file
* New generated bindata.go
* Changes after npm install
* Add argument for cluster configuration file
* Add test for dynamic cluster loading
Not yet sure with name "cluster config".
* Solved conflicts
* Renaming of static peers
* Remove static peers population
* Missing argument for config
* Renaming of static peers to boot nodes for consistency
* Fix of name change
* Cluster config is now cluster data
* Load static nodes from configuration
* Final renaming of var for file content
* Update project to use Whisper v6. Part of #638
* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.
* Enable light mode on Whisper v6 for non-mail servers. Part of #638
* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)
* Update bootnode addresses in staticnodes.json. Part of #638
* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638
* Move MakeTestNodeConfig to utils. Part of #638
* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
Fix failing tests for users w/o access to the ACCOUNT_PASSWORD env variable. Disable e2e public network tests in Travis pull requests. Exclude lib dir from unit tests.
* Accounts are now encrypted with a strong passphrase.
* The encrypted passphrase is stored in .travis.yml
* Public testnet tests are now run after tests on a private net
This PR refactors CLI API, removes obsolete commands and splits status code into smaller pieces:
* get rid of subcommands API (no ./status <command>)
* get rid of custom cli app package
* use stdlib flag package for handling command line flags
* move cross-compilation / mobile related code to lib/ package
* move wnode command into separate binary (cmd/node-status, name is subject to discuss)
* remove faucet command as obsolete
* update/add docs/READMES/wikis for new command line flags
It makes statusd code much simpler and smaller, separates concerns (lib, wnode and statusd are different things).
Refactor and little bit clean up Notify api:
Created interface and package "notification" and extracted related code into it
Set dependencies into constructor notificatotion.Manager
Mocks for notificatotion.Manager and FCMClient
Bacis unit tests for Notify and Send
Notify func is now deprecated
Notify users uses new API: message, Payload and a list of tokens
1. A new private testing network (StatusChain) is introduced.
2. An example test case implemented (TestSendEtherOnStatusChainTx)
3. Genesis file added static/config/status-chain-genesis.json
Geth js commands coming through jail with a callback will now be executed truly asynchronously blocking jail only when an actual interaction with VM is performed.
Technically, it registers a new handler jeth.sendAsync which executes functions with callbacks asynchronously.
Changes include:
1. Send and SendAsync now use cell.VM instead of otto.Otto providing proper locking.
2. Unmarshalling in ExecuionPolicy.ExecuteWithClient is now done into var result interface{} instead of var result json.RawMessage because test case 0 of TestJailWhisper failed providing byte codes instead of 5.0.
3. Due to the asynchronous nature of web3 calls new weird timeouts in tests have been introduced. They may fail sometimes but I gave up trying to implement a more reliable and readable solution.
* static: updates Whisper test (to work with Geth 1.6.1)
* jail: VM persistence implemented
* jail: sendMessage/showSuggestions minor fixes (to be squashed)
* node: CHT and boot nodes auto-load implemented
* Replaced CHT data file from farazdagi's to tiabc's
* Rewrote config_test.go using testify having reduced it twice in size
* Increased SyncTime and panic timeout in tests
* Fixed test - remove go default test to testify/suite (#207)
* Add flag setup for RPCEnabled and add comment (#225)
* jail: register method handlers before running initial js in jail (#226)
* Console Jail Mod #179 (#228)
* Added ./statusd-data into .gitignore
* Increased log level for the test node from INFO to ERROR
* Add call to loop.Run to evaluate all setTimeout/setIntervals methods. (#208)
* Rebase onto geth1.6.7 (#232)
* Got back sync duration from 60s to 30s, updated bindata.go
- node: signals and node reset, fixes#152
- tests update (testify is used)
- node manager refactored, race conditions fixed
- node wrapper has been removed, we rely on go-ethereum node now
- tx queue refactored
- jail refactored
- backend and API introduced
Pascal Precht
Andrea Maria Piana
Vitaliy Vlasov
Richard Ramos
Samuel Hawksby-Robinson
osmaczko
Parvesh Monu
Roman Volosovskyi
Jakub Sokołowski
Jonathan Rainville
Brian Sztamfater
Brian Sztamfater
Pascal Precht
Shivek Khurana
Volodymyr Kozieiev
Adam Babik
yenda
André Medeiros
Adam Babik
Dmitry
Pedro Pombeiro
Kishan Sagathiya
Ivan Daniluk
Dmitry Shulyak
Adam Babik
Alex Kohler
Frank Mueller
Igor Mandrigin
Daniel Nephin
Evgeny Danilenko
perissology
Ewetumo Alexander
Ivan Tomilov
b00ris
screwyprof
Victor Farazdagi