Commit Graph

185 Commits

Author SHA1 Message Date
Andrea Maria Piana fd49b0140e
Move to protobuf for Message type (#1706)
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`

* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.

* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.

* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
2019-12-05 17:25:34 +01:00
Andrea Maria Piana b27779aa4e
Add Mailserver Request Gaps service (#1596)
This commits add some endpoints to store mailserver request gaps,
and provides 4 endpoints:

```
AddMailserverRequestGaps(gaps []MailserverRequestGap) error
GetMailserverRequestGaps(chatID string) []MailserverRequestGap
DeleteMailserverRequestGaps(ids []string) error
DeleteMailserverRequestGapsByChatID(id string) error
```
2019-09-04 12:04:17 +02:00
Adam Babik d7f454fe4c
Add mailservers service (#1595) 2019-09-04 08:25:33 +02:00
Adam Babik e93d994460 integrate status-protocol-go 2019-07-24 18:46:43 +02:00
Adam Babik dbaf622e12
Refactor messaging (#1510)
This change flattens messaging/chat package. It also removes dependency between multidevice and chat/protobuf packages.

The Publisher interface was also changed a bit to support more native types.

Version got bumped to 0.29.0-beta.3.
2019-07-03 21:13:11 +02:00
Adam Babik e28d4ef1a3
Create messaging package and move some packages into it (#1505) 2019-06-26 20:17:41 +02:00
Andrea Maria Piana 5335a2b4fd
Move installations to status-go (#1499)
* Move installations to status-go

This commit moves installations management/storage to status-go.
We remove the native binding and provide RPC endpoints to set the
metadata and return a list of our own installations.
2019-06-26 11:32:59 +02:00
Andrea Maria Piana 0ade9a6cbb
Cache keys (#1497)
* Cache keys

Generating a symkey can take up to a second on slow devices, this commit
makes so that keys are saved once generated and stored in the database.
2019-06-24 09:26:25 +02:00
Andrea Maria Piana f6fba1d3d6 Publish contact code periodically 2019-06-19 17:38:45 +02:00
Andrea Maria Piana 1aa3e2812a Add support for partitioned topic 2019-06-19 17:38:45 +02:00
Andrea Maria Piana 78ed35d2fe Add protocol version to bundle 2019-06-19 17:38:45 +02:00
Andrea Maria Piana cef7f367ab Add topic negotiation
This commit add topic negotiation to the protocol.

On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.

This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).

When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.

This change is backward compatible, as long as there is no downgrade of
the app on the other side.

A few changes:

* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).

* Factored out multidevice management in a separate namespace

* Moved all the test to test the whole protoocl rather than just the encryption service

* Moved all the filter management in status-go
2019-06-19 17:38:45 +02:00
Adam Babik c9e99c432d migrate to go 1.12 and go modules 2019-06-12 13:12:00 +02:00
Andrea Maria Piana 4ab08629f6 Add postgres
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
2019-05-15 11:01:34 +02:00
Andrea Maria Piana 3292538b41
Allow multiple bundles, change n devices & bundle refresh (#1331)
Change to support sending multiple bundles, as needed for group chats,
limit number of devices to 3 as already done in the UI and refresh
bundle daily.
2018-12-21 11:07:25 +01:00
Andrea Maria Piana 1f6cccd0fc
Add enabling/disabling of installations (#1264)
This commit adds a list new table, installations, which is used to keep
track of which installation are active for a given identity key.

In general, we limit the number of installation that we keep
synchronized to 5, to avoid excessive usage of resources.

Any installation coming from our own identity, will have to be manually
enabled, otherwise we trust the other peer has correctly paired their
devices.

We use a timestamp to decide which installations to keep synchronized as
a logical clock would have make the creation of the bundle more
complicated, but this can always be converted to a logical clock at
later stages without breaking compatibility.
2018-11-06 09:05:32 +01:00
Andrea Maria Piana ee3c05c79b
Change handling of skipped/deleted keys & add version (#1261)
- Skipped keys

The purpose of limiting the number of skipped keys generated is to avoid a dos
attack whereby an attacker would send a large N, forcing the device to
compute all the keys between currentN..N .

Previously the logic for handling skipped keys was:

- If in the current receiving chain there are more than maxSkip keys,
throw an error

This is problematic as in long-lived session dropped/unreceived messages starts
piling up, eventually reaching the threshold (1000 dropped/unreceived
messages).

This logic has been changed to be more inline with signals spec, and now
it is:

- If N is > currentN + maxSkip, throw an error

The purpose of limiting the number of skipped keys stored is to avoid a dos
attack whereby an attacker would force us to store a large number of
keys, filling up our storage.

Previously the logic for handling old keys was:

- Once you have maxKeep ratchet steps, delete any key from
currentRatchet - maxKeep.

This, in combination with the maxSkip implementation, capped the number of stored keys to
maxSkip * maxKeep.

The logic has been changed to:

- Keep a maximum of MaxMessageKeysPerSession

and additionally we delete any key that has a sequence number <
currentSeqNum - maxKeep

- Version

We check now the version of the bundle so that when we get a bundle from
the same installationID with a higher version, we mark the previous
bundle as expired and use the new bundle the next time a message is sent
2018-11-05 20:00:04 +01:00
Andrea Maria Piana e4ba365b8a
Add bundles.added signal & pairing endpoint (#1237) 2018-10-16 12:31:05 +02:00
Andrea Maria Piana dcaf8caed0
Add x3dh key exchange (#1127)
* Add x3dh key exchange

* Encrypt using the double ratchet

* Multi device with auto-pairing

* Add pfs enabled flag
2018-09-24 20:07:34 +02:00
Adam Babik 8aef7c4f15
tune CLI for better docs experience (#1207) 2018-09-21 16:09:31 +02:00
Dmitry a11d970112 Update rendezvous nodes accordingle to our inventory
Rendezvous node identities were changed. Probably, previous private keys were
removed due to replacament of the VM. Additional node is from HK DC.
2018-09-20 09:46:59 +03:00
Pedro Pombeiro 99d7bc207d
Only keep two random static nodes from fleets.status.im (#1196) 2018-09-19 17:08:54 +02:00
Pedro Pombeiro 4e7ae20eae
Add script to update fleets static JSON. Closes #1188 (#1189) 2018-09-14 10:46:34 +02:00
Pedro Pombeiro 3d00af7fa3
Streamline configuration in status-go. Part of #1180 (#1183)
- Replace command line flags with `-c` config flag. Part of #1180
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
2018-09-13 18:31:29 +02:00
Kishan Sagathiya 0f5ad2545d Issue #765 Read configuration without go-bindata (#1158)
In the current codebase if you want to use different keys (or any
other configuration) for tests it is required to place them at a
certain path in static folder, bundle them with go-bindata and only
then run tests. This is simply unnecessary.

This commit instead directs loads configurations from file without
relying on go-bindata
2018-08-21 14:46:10 +02:00
Pedro Pombeiro ebc77374b2 Remove unused `test.sol` file
- File is no longer used after jail code was removed
- Ensure that `go-bindata` is set up on `make setup`
2018-08-15 14:52:40 +02:00
Ivan Daniluk 801053a17d
Remove jail package (#1106) 2018-07-24 12:05:21 +02:00
Dmitry Shulyak b120587c9b Update bindata 2018-04-09 15:42:36 +02:00
Dmitry Shulyak 88f881c83e
Rewrite config as a native golang structure and remove it from static (#762) 2018-03-26 15:52:20 +03:00
Adam Babik 8b94ac00bb
Update mainnet dev cluster config (#745) 2018-03-22 15:58:28 +01:00
Alex Kohler 4a01a395b4 Move status-go over to go-ethereum/log #717 (#727) 2018-03-20 19:35:28 +01:00
Frank Mueller dc4a0a98ed Fix setting of bootstrap nodes (#744)
Fix dropped bootstrap nodes setting and rename of cluster config and separation of nodes
2018-03-20 16:05:21 +02:00
Frank Mueller 8e42014671
Static peers are now also optionally configurable (#738)
* Rename bootnode to static peers

Also add some groupings between types.

* Remove not needed genesisHash

* More cleanup of bootnodes with static peers

* Add option of cluster configuration file

* New generated bindata.go

* Changes after npm install

* Add argument for cluster configuration file

* Add test for dynamic cluster loading

Not yet sure with name "cluster config".

* Solved conflicts

* Renaming of static peers

* Remove static peers population

* Missing argument for config

* Renaming of static peers to boot nodes for consistency

* Fix of name change

* Cluster config is now cluster data

* Load static nodes from configuration

* Final renaming of var for file content
2018-03-19 17:22:09 +01:00
Pedro Pombeiro d84510de92
Remove unused JavaScript tests for RPC API (#737) 2018-03-16 18:49:42 +01:00
Pedro Pombeiro 38e779ef22
Remove mailservers from staticpeers.json, since connections are limited (#733) 2018-03-14 14:50:45 +01:00
Pedro Pombeiro ec83efc26a
Add two Whisper v6 full nodes to `staticpeers.json` (#721) 2018-03-02 16:48:45 +01:00
Pedro Pombeiro 6cdea4ef97
Update codebase to leverage Whisper v6 (#703)
* Update project to use Whisper v6. Part of #638

* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.

* Enable light mode on Whisper v6 for non-mail servers. Part of #638

* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)

* Update bootnode addresses in staticnodes.json. Part of #638

* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638

* Move MakeTestNodeConfig to utils. Part of #638

* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
2018-03-02 10:25:30 +01:00
Adam Babik 0a15c9ff9a
Configure mainnet nodes (#701)
Add mainnet static peers to static/config/staticpeers.json.
2018-02-26 19:36:04 +01:00
Pedro Pombeiro 134d01fcc1
Fix incorrect path in Makefile and run `make generate`. Part of #692 2018-02-22 12:19:31 +01:00
Igor Mandrigin c06d58addd Remove `notifications` package from go-ethereum. 2018-02-22 11:10:21 +01:00
Adam Babik 89d89681a3
Add support for metrics (#558)
It adds support for metrics (expvar and Prometheus) along with docker-compose files to run a Whisper test cluster.
2018-01-30 12:51:48 +01:00
Daniel Nephin 755a646bd4 Use gometalinter config instead of makefile (#560) 2018-01-24 00:25:28 -08:00
Ivan Daniluk ea55ac0744
Cleanup after merging wnode-status and statusd (#554) 2018-01-17 22:40:14 +01:00
Ivan Daniluk f93cd81d83
Upgrade to geth 1.7.3 and add geth patches (#492) 2018-01-15 21:26:41 +01:00
Adam Babik b7fb51d92a
Refactor shh_requestMessages method (#513) 2017-12-21 11:26:01 +01:00
Evgeny Danilenko 919ab285d3 add password for MailServer encryption (#494) 2017-12-07 17:58:11 +01:00
Adam Babik ac76e4731f
Prepare wnode-status to support MailServer (#477) 2017-12-04 17:11:14 +01:00
Adam Babik d6d5945ab4
Clean up Jenkinsfile-manual and Travis (#478)
Small clean up of Jenkinsfile-manual required after rebuilding Jenkins. Also improved .travis.yml a bit.
2017-11-29 07:48:40 +01:00
perissology f0beeb3829 fix failing tests in PRs from other repos, fixes #459 (#461)
Fix failing tests for users w/o access to the ACCOUNT_PASSWORD env variable. Disable e2e public network tests in Travis pull requests. Exclude lib dir from unit tests.
2017-11-20 10:21:30 -08:00
Evgeny Danilenko 209d5fed74 Issue/fix public testnet tests #424 (#438)
* Accounts are now encrypted with a strong passphrase.
* The encrypted passphrase is stored in .travis.yml
* Public testnet tests are now run after tests on a private net
2017-11-07 12:46:11 -05:00
Ewetumo Alexander cb5ccb52c4 Emit messages logs for processing and sorting out messaging problems (#420)
* Add DeliveryService to Whisper to track incoming and outgoing messages.
* Have log tagged log lines for incoming and outgoing messages.
2017-11-06 14:10:52 -05:00
Ivan Daniluk 4536e99275 Improve statusd CLI usage (#441)
This PR refactors CLI API, removes obsolete commands and splits status code into smaller pieces:

* get rid of subcommands API (no ./status <command>)
* get rid of custom cli app package
* use stdlib flag package for handling command line flags
* move cross-compilation / mobile related code to lib/ package
* move wnode command into separate binary (cmd/node-status, name is subject to discuss)
* remove faucet command as obsolete
* update/add docs/READMES/wikis for new command line flags

It makes statusd code much simpler and smaller, separates concerns (lib, wnode and statusd are different things).
2017-11-03 18:07:13 -04:00
Adam Babik 987cdd6221 Update Ropsten CHT to number 478 (#428) 2017-10-29 16:44:44 +03:00
Ivan Tomilov 2897f0ec0f Merge pull request #390 from status-im/issue/refactor-api-notify-send-messages-#342
Refactor and little bit clean up Notify api:

Created interface and package "notification" and extracted related code into it
Set dependencies into constructor notificatotion.Manager
Mocks for notificatotion.Manager and FCMClient
Bacis unit tests for Notify and Send
Notify func is now deprecated
Notify users uses new API: message, Payload and a list of tokens
2017-10-26 20:54:12 +03:00
Adam Babik 71c259dd95 turn off vet shadow 2017-10-26 13:10:56 +02:00
Adam Babik d058b9105a remove goimports 2017-10-26 12:50:30 +02:00
b00ris 22fb4dfc30
integrate linters into travis 2017-10-23 13:28:48 +03:00
Eugene 4aaeeb6ebf
Extract notification into separete package and interface 2017-10-22 20:39:51 +03:00
b00ris e911666b5d Fix make lint warnings (#417)
fixed make lint warnings
cleared linter_exclude_list.txt
removed some commented code
fixed comments from #388
2017-10-20 12:06:22 +03:00
Adam Babik ad9a8777a8 fix node unit tests (#416)
Fix tests that broke after introducing #414
2017-10-19 16:45:14 +03:00
Adam Babik 2e9302c631 removed CHT root from Rinkeby (#414)
CHT roots for Rinkeby do not work as their presence prevents status-go from even starting synchronization.
2017-10-19 15:02:42 +03:00
screwyprof 26fcfda87c Introduced a private Ethereum network for testing purposes (#397)
1. A new private testing network (StatusChain) is introduced.
2. An example test case implemented (TestSendEtherOnStatusChainTx)
3. Genesis file added static/config/status-chain-genesis.json
2017-10-17 00:54:56 +03:00
Ivan Tomilov 88294b29be Removed homestead bootnodes (#394)
Remove homestead bootnodes from cht.json. Their existence reflected the reality incorrectly as we don't have any homestead bootnodes in our cluster.
2017-10-12 12:46:41 +03:00
Adam Babik 90acfedf7a Rebase geth 1.7.0 (#353) 2017-10-10 12:38:49 +03:00
Ivan Tomilov fc8f59e121 Fixed eth_sendTransaction routing to the local node (#351)
* Fixed eth_sendTransaction routing to the local node
* Add local RPC handlers for eth_accounts and eth_sendTransaction
2017-09-25 18:04:40 +02:00
Ivan Tomilov 5f19c9cd0a Implemented sendAsync for js commands with a callback (#321)
Geth js commands coming through jail with a callback will now be executed truly asynchronously blocking jail only when an actual interaction with VM is performed.

Technically, it registers a new handler jeth.sendAsync which executes functions with callbacks asynchronously.

Changes include:

1. Send and SendAsync now use cell.VM instead of otto.Otto providing proper locking.
2. Unmarshalling in ExecuionPolicy.ExecuteWithClient is now done into var result interface{} instead of var result json.RawMessage because test case 0 of TestJailWhisper failed providing byte codes instead of 5.0.
3. Due to the asynchronous nature of web3 calls new weird timeouts in tests have been introduced. They may fail sometimes but I gave up trying to implement a more reliable and readable solution.
2017-09-18 15:13:32 +03:00
Ivan Tomilov c85525e139 Extended testGetDefaultConfig to support Rinkeby (#241)
This pull requests only extends testGetDefaultConfig to run against Rinkeby as well and fixes TestJailWhisper.
2017-09-08 15:32:02 +03:00
Ivan Tomilov 9463d3843a Updated CHT (#279)
Syncing for Ropsten should now start from something around block 1617921 and from 839680 for Rinkeby.
2017-09-07 10:54:33 +03:00
Adam Babik a4d2fdfa4a cht: remove downloading CHT from GH but embed it (#270) 2017-08-25 10:56:54 +01:00
Ivan Tomilov ebd77aabe2 Merging bug/whisper-on-geth1.6.1 (#236) which acts like develop
* static: updates Whisper test (to work with Geth 1.6.1)
* jail: VM persistence implemented
* jail: sendMessage/showSuggestions minor fixes (to be squashed)
* node: CHT and boot nodes auto-load implemented
* Replaced CHT data file from farazdagi's to tiabc's
* Rewrote config_test.go using testify having reduced it twice in size
* Increased SyncTime and panic timeout in tests
* Fixed test - remove go default test to testify/suite (#207)
* Add flag setup for RPCEnabled and add comment (#225)
* jail: register method handlers before running initial js in jail (#226)
* Console Jail Mod #179 (#228)
* Added ./statusd-data into .gitignore
* Increased log level for the test node from INFO to ERROR
* Add call to loop.Run to evaluate all setTimeout/setIntervals methods. (#208)
* Rebase onto geth1.6.7 (#232)
* Got back sync duration from 60s to 30s, updated bindata.go
2017-08-04 23:14:17 +07:00
Victor Farazdagi bc3ea62eca config: add CHT root URL, fixes #170 2017-05-27 23:42:03 +03:00
Victor Farazdagi 35eb15fed7 api: make is completely sequential 2017-05-25 15:34:13 +03:00
Victor Farazdagi 349103de1a major refactoring: node manager, tx queue, accounts, jail
- node: signals and node reset, fixes #152
- tests update (testify is used)
- node manager refactored, race conditions fixed
- node wrapper has been removed, we rely on go-ethereum node now
- tx queue refactored
- jail refactored
- backend and API introduced
2017-05-24 00:29:20 +03:00
Victor Farazdagi b0c3489d22 params: ropsten dev/prod config updated, fixes #167 2017-05-17 15:47:39 +03:00
Victor Farazdagi 82fe888853 whisper5: minor fixes 2017-05-16 11:02:34 +03:00
Victor Farazdagi 4f27a79d83 config: homestead and rinkeby confg + allow dev/prod modes 2017-05-16 10:51:11 +03:00
Victor Farazdagi 88394690c2 cmd/statusd: faucet sub-command added, refs #159 2017-05-16 10:51:11 +03:00
Victor Farazdagi 9a7caa6b58 config: expose bootstrap cluster config, closes #139 2017-05-16 10:51:11 +03:00
Victor Farazdagi 68d4d20d66 cmd/statusd, geth: VerifyAccountPassword method exposed, closes #151 2017-05-15 13:48:27 +03:00
Victor Farazdagi 2f0c93fd3b refactor: linters + recommendations applied, closes #146 2017-05-15 08:19:46 +03:00
Victor Farazdagi 8876fd04d3 whisper: JavaScript tests to cover updated Whisper API 2017-05-14 02:34:08 +03:00
Victor Farazdagi 92afd0d47e vendor/whisper, statusd: push notifications implemented, closes #135 2017-05-03 16:09:48 +03:00
Victor Farazdagi ecea845d88 rebase: minor fixes to integrate with Geth 1.6.1 2017-05-03 16:00:36 +03:00
Victor Farazdagi 9a68fd211f whisper: `statusd wnode` sub-command, closes #140 2017-04-28 11:50:12 +03:00
Victor Farazdagi 21344af199 geth, jail: embed static resources, closes #131 2017-04-28 11:50:12 +03:00