Commit Graph

156 Commits

Author SHA1 Message Date
Samuel Hawksby-Robinson 07e46714f0
Anon Metrics Broadcast (#2198)
* Protobufs and adapters

* Added basic anon metric service and config init

* Added fibonacci interval incrementer

* Added basic Client.Start func and integrated interval incrementer

* Added new processed field to app metrics table

* Added id column to app metrics table

* Added migration clean up

* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests

There was a wierd bug where metrics in the db that did not explicitly insert a  value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.

* Updated adaptors and db to handle proto_id

I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.

* Moved incrementer into dedicated file

* Resolve incrementer test fail

* Finalised the main loop functionality

* Implemented delete loop framework

* Updated adaptors file name

* Added delete loop delay and quit, and tweak on RawMessage gen

* Completed delete loop logic

* Added DBLock to prevent deletion during mainLoop

* Added postgres DB connection, integrated into anonmetrics.Server

* Removed proto_id from SQL migration and model

* Integrated postgres with Server and updated adaptors

* Function name update

* Added sample config files for client and server

* Fixes and testing for low level e2e

* make generate

* Fix lint

* Fix for receiving an anonMetricBatch not in server mode

* Postgres test fixes

* Tidy up, make vendor and make generate

* delinting

* Fixing database tests

* Attempted fix of does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error on sql resource loas

* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work

* Lint for the lint god

* Why doesn't the linter list all its problems at once?

* test tweaks

* Fix for wakuV2 change

* DB reset change

* Fix for postgres db migrations fails

* More robust implementation of postgres test setup and teardown

* Added block for anon metrics functionality

* Version Bump to 0.84.0

* Added test to check anon metrics broadcast is deactivated

* Protobufs and adapters

* Added basic anon metric service and config init

* Added new processed field to app metrics table

* Added id column to app metrics table

* Added migration clean up

* Added appmetrics GetUnprocessed and SetToProcessedByIDs and tests

There was a wierd bug where metrics in the db that did not explicitly insert a  value would be NULL, so could not be found by . In addition I've added a new primary id field to the app_metrics table so that updates could be done against very specific metric rows.

* Updated adaptors and db to handle proto_id

I need a way to distinguish individual metric items from each other so that I can ignore the ones that have been seen before.

* Added postgres DB connection, integrated into anonmetrics.Server

* Removed proto_id from SQL migration and model

* Integrated postgres with Server and updated adaptors

* Added sample config files for client and server

* Fix lint

* Fix for receiving an anonMetricBatch not in server mode

* Postgres test fixes

* Tidy up, make vendor and make generate

* Moved all anon metric postgres migration logic and sources into a the protocol/anonmetrics package or sub packages. I don't know if this will fix the does:  cannot open `does' (No such file or directory)
not:   cannot open `not' (No such file or directory)
exist: cannot open `exist' (No such file or directory) error that happens in Jenkins but this could work
2021-09-01 13:02:18 +01:00
Andrea Maria Piana 2c96475aac [Fixes: #2328] Fixes iterator on leveldb
Leveldb did not support topics only queries.
This commit changes the behavior so that now we use the topics map if
present.
2021-08-24 15:40:40 +02:00
Andrea Maria Piana 4b0daeb47b Move services to status-node
Move all the services to status-node, as some of them were there, some
of them were in the geth backend and scattered around.
2021-07-20 10:57:38 +02:00
Andrea Maria Piana d092a2bb49 Fix wallet big int conversion 2021-07-20 10:57:38 +02:00
RichΛrd 40359f9c1b
go-waku integration (#2247)
* Adding wakunode module
* Adding wakuv2 fleet files
* Add waku fleets to update-fleet-config script
* Adding config items for waku v2
* Conditionally start waku v2 node depending on config
* Adapting common code to use go-waku
* Setting log level to info
* update dependencies
* update fleet config to use WakuNodes instead of BootNodes
* send and receive messages
* use hash returned when publishing a message
* add waku store protocol
* trigger signal after receiving store messages
* exclude linting rule SA1019 to check deprecated packages
2021-06-16 16:19:45 -04:00
Andrea Maria Piana d50fee6bb2 Handle connection state 2021-05-21 07:22:58 +02:00
Andrea Maria Piana 57b1bc193f Remove whisper 2021-05-21 07:22:58 +02:00
Andrea Maria Piana cf6ef3171a Add mailserver logic 2021-05-21 07:22:58 +02:00
Volodymyr Kozieiev 09942bf200 Confirmation settings now correctly processed on mailserver 2021-03-01 11:23:17 +01:00
Andrea Maria Piana a1b3e3a772
Clean topics that we don't listen to
There was a bug on status-react where it would save filters that were
not listened to.
This commit adds a task to clean up those filters as they might result
in long syncing times.

This commit also returns topics/ranges/mailserves from messenger in
order to make the initialization of the app simpler and start moving
logic to status-go.

It also removes whisper from vendor.
2021-01-26 09:39:57 +01:00
Gheorghe Pinzaru d04e54e54e
Local notifications service (#2026)
Also adds implementation for eth transactions notifications
2020-10-28 10:56:14 +03:00
Andrea Maria Piana e836ffb47f Soft blacklist peer ids
Why make this change?

This change is useful so we can blacklist peer ids, without actually
disconnecting them, so that it's harder for them to tell that they are
being blacklisted. Envelopes will just be dropped.

What has changed?

In waku we pass a config with a list of hex-encoded peer ids, and we
check that against newly received envelopes.
If so we just drop the envelope and return
2020-09-09 22:57:15 +02:00
Andrea Maria Piana d7222c89e0
shuffle servers and limit & filter by chat id 2020-08-27 18:54:28 +02:00
Andrea Maria Piana 401d18e558
Enabled push notifications by default and add fleet 2020-08-27 18:54:06 +02:00
Andrea Maria Piana d09eaeacb8 Allow node to start a push notification server
This commit allows a node to start a push notification server.
If the config is set it will start a messenger with a corresponding pn
server.
2020-08-20 10:54:33 +02:00
Jakub Sokołowski ea47f07b96 make Waku the default protocol
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-08-20 10:27:47 +02:00
Andrea Maria Piana c5577418b7 Implement rate limiting by bytes
Why make this change?

Currently we only rate limit by number of messages.
This works well if we want to limit a large amount of small messages but
breaks down if sending a smaller amount of large messages.

This commit extends the current code to limit by size as well, setting
the default to 1MB per second, which should be enough.

What has changed?

- Rate limiter for waku only
2020-06-15 19:46:23 +02:00
Andrea Maria Piana 2d12ac4fbb Rename rate limiting to packet rate limiting 2020-06-15 19:46:23 +02:00
Andrea Maria Piana 0bffeab908 Address feedback from code review and add quoted image 2020-06-01 11:30:04 +02:00
Andrea Maria Piana d5086d6e89 Remove incentivisation service
Incentivisation was an experiment in running an incentivised fleet that
rewarded nodes based on their well behavior. It was heavily influenced
by https://docs.loki.network/ . It is currently not used anymore, so
removing.
2020-06-01 11:30:04 +02:00
Andrea Maria Piana 3e312cb167 Lower default MinimumPoW
This commit lowers the default minimum pow for both Waku & Whisper.
We need to override the value as mobile clients have this already stored
in the database.

This is done in order to support sending of images.
2020-06-01 11:30:04 +02:00
Adam Babik 6fdeb9b115
Fix listen flag (#1897) 2020-03-10 13:40:35 +01:00
Andrea Maria Piana 4d2ecb0055 Enable eth config 2020-02-28 18:47:25 +01:00
Adam Babik b33ad8147d
Add example to demonstrate Whisper-Waku bridge (#1875) 2020-02-26 20:36:25 +01:00
Adam Babik f335d1cf7e
Prevent starting node without APIModules (#1876) 2020-02-26 20:35:47 +01:00
Andrea Maria Piana 849492fda9
Add statusUpdate code (#1861)
* Add status-option code

This commits changes the behavior of waku introducing a new status-code,
`2`, that replaces the current single options codes.

* linting
2020-02-21 15:48:53 +01:00
Jakub Sokołowski 86f54c3f5f remove eth.beta fleet as it has been decomissioned
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-02-21 14:48:47 +01:00
Adam Babik 76b5dc29dc
Integrate Whisper-Waku bridge in status-go (#1854) 2020-02-18 12:21:01 +01:00
Jakub Sokołowski 6b5e97d148 statusd: use eth.prod fleet by default
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2020-02-11 16:11:41 +01:00
Adam Babik 8b61d92d58
Remove protocol and eth-node submodules (#1835) 2020-02-10 12:22:37 +01:00
Andrea Maria Piana f2eebd1e49
Verify ENS in the background (#1824)
Currently ENS are verified explicitly by status-react, this is not ideal
as if that fails it will have to be explicilty retried in status-react.
This commits changes that behavior so that ENS are verified in a loop
and updated if new messages are received.
2020-02-05 11:09:33 +01:00
Roman Volosovskyi 9cf640842b
[status-im/status-react#9942] Upgradable paths in configs
Storing absolute path for different configs breaks compatibility on iOS
as app's dir is changed after upgrade. The solution is to store relative
paths and to concatenate it with `backend.rootDataDir`. The only
exception is `LogFile` as it is stored outside `backend.rootDataDir` on
Android. `LogDir` config was added to allow adding of custom dir for log
file.
Configs concerned:
`DataDir`
`LogDir`
`LogFile`
`KeystoreDir`
`BackupDisabledDataDir`
2020-02-03 20:02:01 +02:00
Andrea Maria Piana 25d46c6d82
Fix waku tests & contact ens (#1802) 2020-01-17 13:39:09 +01:00
Andrea Maria Piana 88a1d0111e
Add Commands (#1731)
This commit adds handling of Transaction commands.
2020-01-10 19:59:01 +01:00
Adam Babik 37a508a97b
Integrate Waku service (#1759) 2020-01-08 12:12:23 +01:00
Pedro Pombeiro c8a911ebd1 Use goimports instead of gofmt 2020-01-06 10:17:23 +01:00
Pedro Pombeiro b99af530e1 Migrate more files to eth-node/crypto 2019-12-20 11:24:26 +01:00
Adam Babik 4f45bceb9a
Drop peers when rate limit is exceeded (#1732) 2019-12-16 20:38:50 +01:00
Adam Babik a636f33109
Set Whisper version submodule to v6 (#1725) 2019-12-09 11:36:14 +01:00
Adam Babik 4ac4a61e20
Move whisper to status-go monorepo (#1720) 2019-12-09 11:06:04 +01:00
Adam Babik c199c8f342
Update Whisper and introduce rate limiting (#1673) 2019-11-18 11:22:23 +01:00
Jakub Sokołowski 74e553b847 add test to check that version doesn't start with v
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-10-29 12:53:27 +01:00
Jakub Sokołowski 98c861cfe2 make sure Version is always set in loadNodeConfig()
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-10-29 12:53:27 +01:00
Adam Babik d7f454fe4c
Add mailservers service (#1595) 2019-09-04 08:25:33 +02:00
Andrea Maria Piana 9de77b21b2 Add datasync, v1messages & disable discovery topic options
Adds support for datasync, V1Messages and disabling the discovery topic.
This is a backward compatible change as long as they are not toggled on
(they are not by default).
2019-07-29 19:39:44 +02:00
Dmitry Shulyak 9723b64827
Permissions api (#1524)
* Add permissions api

* Integrate permissions service

* Reduce cyclomatic complexity of the MakeNode function
2019-07-21 08:41:30 +03:00
Dmitry Shulyak 9dbf5a0c86
Add browsers API (#1523)
Add browsers API
2019-07-17 08:28:37 +03:00
Andrea Maria Piana 1aa3e2812a Add support for partitioned topic 2019-06-19 17:38:45 +02:00
Andrea Maria Piana cef7f367ab Add topic negotiation
This commit add topic negotiation to the protocol.

On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.

This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).

When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.

This change is backward compatible, as long as there is no downgrade of
the app on the other side.

A few changes:

* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).

* Factored out multidevice management in a separate namespace

* Moved all the test to test the whole protoocl rather than just the encryption service

* Moved all the filter management in status-go
2019-06-19 17:38:45 +02:00
Dmitry Shulyak 047c9b5263
Download transfers starting from latest block header (#1467) 2019-06-14 13:16:30 +03:00
Andrea Maria Piana 4ab08629f6 Add postgres
This commits adds support for postgres database.
Currently two fields are stored: the bloom filter and the topic.
Only the bloom filter is actually used to query, but potentially we will
use also the topic in the future, so easier to separate it now in order
to avoid a migration.
2019-05-15 11:01:34 +02:00
Dmitry Shulyak 218a35e609
Notify users that envelope was discarded and retry sending it (#1446)
API for querying mail servers that skips already received data on subsequent requests
2019-04-30 09:46:12 +03:00
Andrea Maria Piana 354e6981ba
Add network incentivisation service (#1452) 2019-04-29 14:05:49 +02:00
Jakub Sokołowski 1a4fe50971 add a short doc about config.json
Signed-off-by: Jakub Sokołowski <jakub@status.im>
2019-04-25 16:20:11 +02:00
Dmitry eeaf669006 Add goerli testnet 2019-03-19 10:56:09 +02:00
Dmitry 5bce7e793f Redirect log output to mobile system loggers 2019-03-07 17:07:12 +02:00
Dmitry Shulyak cf8f20b23c Use lumberjack library to rotate logs and add config that enables it (#1399) 2019-03-01 14:37:13 +01:00
Dmitry Shulyak aa28f652e3 Re-send user payload if previously sent envelope wasn't acknowledged (#1386)
* Split shhext.tracker into envelopes and mail monitors

* Send envelopes on every new attempt to deliver a message

* Re-send user payloads if previous envelopes weren't acknowledged

* Remove debug api across the codebase
2019-03-01 14:36:21 +01:00
Dmitry c8a616688c Add request with retries api call 2019-01-24 14:31:56 +02:00
Adam Babik d20b5dc3b3
mailserver: build pruning into WMailServer (#1342)
MailServer pruning was implemented as a separate command but that required stopping a mail server and executing the command manually. This change builds pruning into MailServer and can be set using MailServerDataRetention in WhisperConfig.
2019-01-23 14:32:45 +01:00
Dmitry 1a365a5140 Require BackupDisabledDataDir only if PFSEnabled is true 2019-01-22 16:56:35 +02:00
Dmitry aa09d854ca Prevent frequent requests with same topics 2019-01-18 14:52:33 +02:00
Dmitry Shulyak 863d07f798 Accept status service config directly from json input (#1337) 2019-01-17 13:56:22 +01:00
Adam Babik 944a35a11b
Validate Whisper DataDir to be relative to the main DataDir (#1335) 2019-01-04 12:44:01 +01:00
Dmitry Shulyak d51761f83e Ensure sent event is received from a mailserver (#1304)
In previous change i used same filtering for a handler that processes confirmations.
But if we are connected with peers that do not support confirmations on whisper level,
whisper may send "Sent" event when envelope is written to a socket. This is our old behaviour.

So, this PR allows to use confirmations from a mail servers and ensure that transition between
multiple version of peers will be smooth.
2018-12-06 10:30:57 +01:00
Adam Babik e658366d1e
Improve Makefile (#1275)
* change docker image tag to RELEASE_TAG
* use changelog in release
2018-11-20 11:13:42 +01:00
Pedro Pombeiro 87f1af0743 Ensure that `NewNodeConfigWithDefaults` builds a valid configuration 2018-11-09 17:02:55 +01:00
b00ris bdbceba6eb
added ulc config params (#1255)
* added ulc config params

* fmt

* fix lint
2018-10-24 19:31:41 +03:00
Adam Babik a75f9c34cf
Request historical messages in batches (#1245) 2018-10-19 11:09:13 +02:00
Adam Babik 20f8f1f2cc
allow configure HTTPVirtualHosts and HTTPCors settings (#1236) 2018-10-12 14:58:32 +02:00
Adam Babik fc54e0b06c
make node-canary working again (#1221)
* removed fleet validation (no fleet is allowed)
2018-10-04 12:40:26 +02:00
Andrea Maria Piana dcaf8caed0
Add x3dh key exchange (#1127)
* Add x3dh key exchange

* Encrypt using the double ratchet

* Multi device with auto-pairing

* Add pfs enabled flag
2018-09-24 20:07:34 +02:00
Adam Babik 8aef7c4f15
tune CLI for better docs experience (#1207) 2018-09-21 16:09:31 +02:00
Adam Babik fd99487328
fix updating peers limit if LES is enabled in CLI (#1199) 2018-09-19 20:29:42 +02:00
Pedro Pombeiro 5b98b8831b
Add peer limits to CLI defaults (#1192) 2018-09-14 12:25:52 +02:00
Pedro Pombeiro 3d00af7fa3
Streamline configuration in status-go. Part of #1180 (#1183)
- Replace command line flags with `-c` config flag. Part of #1180
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
2018-09-13 18:31:29 +02:00
Adam Babik 0136ceb46e
add rendezvous nodes to eth.beta (#1179) 2018-09-04 11:20:58 +02:00
Dmitry Shulyak 5ef26596e4
Enable proxying and discovery of rinkeby nodes (#1172) 2018-08-27 12:32:14 +03:00
Adam Babik 3521c2ac45
Remove NetworkID from cluster (#1163) 2018-08-26 12:54:58 +02:00
Dmitry Shulyak 7cd81a69f7
Add rendezvous nodes and search for 2 les peers if les is enabled (#1167) 2018-08-24 14:20:50 +03:00
Adam Babik 16c7b94eb5
Add labels to docker images and add Jenkinsfile to build docker images (#1160)
* fixed generating `params.Version`
* added labels to all Docker images including git_commit and the author of the build (using `$USER` env variable)
2018-08-24 12:25:07 +02:00
Dmitry Shulyak 6858662d59
Proxy records found in ethereum discovery v5 (#1165) 2018-08-24 06:17:32 +03:00
Adam Babik 19b53030fc
add fleet param to GenerateConfig binding (#1162) 2018-08-21 15:48:58 +02:00
Kishan Sagathiya 0f5ad2545d Issue #765 Read configuration without go-bindata (#1158)
In the current codebase if you want to use different keys (or any
other configuration) for tests it is required to place them at a
certain path in static folder, bundle them with go-bindata and only
then run tests. This is simply unnecessary.

This commit instead directs loads configurations from file without
relying on go-bindata
2018-08-21 14:46:10 +02:00
Andrea Franz 874a3e8151
Feature/mailserver registry smart contract (#1135)
* add verifier and test using simulated backend

* add ContractCaller

* commit simulated backend after deploy and after smart contract writes

* use bind.NewKeyedTransactor for all transactions in tests

* rename RegistryVerifier to Verifier

* initialize contract verifier if MailServerRegistryAddress config is set

* use contractAddress.Hash()

* refactoring

* use fmt.Sprintf to format contract address in logs

* fix test and lint warnings

* update Gopkg.lock

* update Gopkg.lock once more
2018-08-20 15:55:43 +02:00
Adam Babik cf21f981f7
Fix rendezvous loop when discovery protocol is stopped; add advertiseaddr flag (#1151) 2018-08-17 08:25:55 +02:00
Dmitry e935c7de99 Add rendezvous nodes to cluster 2018-08-13 14:37:35 +03:00
Igor Mandrigin a6d69ebae9 Use the same boot & static nodes for Rinkeby as in Mainnet.
ETHIndia and other conference events requires users to be on Rinkeby, hence the users needs to be
able to communicate between Rinkeby/Mainnet.

We used to keep Rinkeby separate, but this commit changes it.
2018-08-10 13:27:43 +02:00
Pedro Pombeiro 9587d030a6 mailserver-canary: Disable fleet specification 2018-08-08 11:21:13 +02:00
Adam Babik 71b8e0e73d
Remove shh and shhext from APIModules (#1131)
Addresses #912. By removing these two modules from APIModules, we disallow DApps from using dangerous methods like shh.SetMinPoW
2018-08-02 08:48:44 +02:00
Adam Babik f0f1b8d783
fix MailServer addresses in cluster config (#1118) 2018-07-26 11:36:04 +02:00
Adam Babik 3b8c6c8260
Add interface to verify trusted MailServers (#1112)
The goal of this PR is to add an interface to verify MailServers. In this PR, MailServers are hardcoded in status-go. The next iteration will use a smart contract.
2018-07-25 16:48:02 +02:00
Adam Babik 408ba5a7e6
Configure eth.staging fleet (#1111)
Provide a way to run a status node with a different fleet.
2018-07-25 16:03:35 +02:00
Dmitry eeca435064 Add rendezvous implementation for discovery interface
Update vendor

Integrate rendezvous into status node

Add a test with failover using rendezvous

Use multiple servers in client

Use discovery V5 by default and test that node can be started with rendezvous discovet

Fix linter

Update rendezvous client to one with instrumented stream

Address feedback

Fix test with updated topic limits

Apply several suggestions

Change log to debug for request errors because we continue execution

Remove web3js after rebase

Update rendezvous package
2018-07-25 15:10:57 +03:00
Adrià Cidre bfbb02019f
Discover mail servers on demand. (#1082)
* [#1076] Discover mail servers on start up.

* [#1076] On-demand mail-server discovery
2018-07-16 09:40:40 +02:00
Jakub 5abc68b11d fix enode IPs for DigitalOcean static whisper nodes (#1095) 2018-07-12 14:58:12 +02:00
Adam Babik 38a60135b2 Add asymmetric key support for MailServer requests (#1075)
* add Asymmetric Key support for MailServer requests

* remove deprecated notice

* fix linter

* refactoring Whisper config related to MailServer

* fix race condition
2018-07-04 11:30:57 +02:00
Adrià Cidre 5bbfabde94
New debug_postSync endpoint (#1046)
* [#ISSUE-1041] New debug_postconfirm endpoint

* move debug service api inside shhext service
2018-06-25 15:27:17 +02:00
Pedro Pombeiro 3b052098fe Update to geth 1.8.11. Closes #1033
- Replace deprecated common.Hex with hexutil.Encode.
- Remove upstreamed 0010-geth-17-fix-npe-in-filter-system.patch.
- Remove upstreamed 0020-discv5-metrics.patch.
- Remove upstreamed 0026-ethdb-error-deadlock.patch.
- Update goleveldb to same version used by geth 1.8.11.
- Update PublicTransactionPoolAPI.GasPrice return type to match type in internal geth interface.
2018-06-22 18:04:13 +02:00
Adrià Cidre 6173d3ee06
[#1022] Make ntp synchronization configuratble (#1026) 2018-06-13 13:24:04 +02:00