Commit Graph

461 Commits

Author SHA1 Message Date
Andrea Franz fa390a52ae
RequestHistoricMessages response (#1009)
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Adrià Cidre 8141657ad7
[#1018] Add a flag to disable shell colors and move verbose logs to debug level (#1020) 2018-06-12 09:49:38 +02:00
Pedro Pombeiro 3f1b2fa57f Update go-duktape.v3 revision to include submitted patch
- Gets rid of noisy build warnings
2018-06-11 15:53:56 +02:00
Adam Babik b5f05b1ab5
Fix flaky TestSingleTopicDiscoveryWithFailover and others (#992)
Other changes:
* needed to patch that loop implementation in Discover V5 implementation in go-ethereum,
* fixed TestStatusNodeReconnectStaticPeers,
* fixed TestBackendAccountsConcurrently.
2018-05-26 09:37:13 +02:00
Dmitry Shulyak ed1cdf0418
Custom status bootnodes (#968)
This change makes invalidation mechanism more aggressive. With a primary goal to invalidate short living nodes faster. In current setup any node that became known in terms of discovery will stay in this state until it will fail to respond to 5 queries. Removing them earlier from a table allows to reduce latency for finding required nodes.

The second change, one adds a version for discovery, separates status dht from ethereum dht.
After we rolled out discovery it became obvious that our boot nodes became spammed with irrelevant nodes. And this made discovery process very long, for example with separate dht discovery takes ~2s, with mutual dht - it can take 1m-10m and there is still no guarantee to find a max amount of peers, cause status nodes is a very small part of whole ethereum infra.

In my understanding, we don't need to be a part of ethereum dht, and lower latency is way more important for us.

Closes: #941
Partially closes: #960 (960 requires futher investigations on devices)
2018-05-18 16:43:07 +03:00
Adam Babik 6da469140f
Refactor in-app metrics (#963)
Remove Prometheus and other metric types and use metrics package from go-ethereum.
2018-05-16 17:36:59 +02:00
Dmitry Shulyak ca962ddaa3 Whisper broadcast loop spams way too hard with debug level 2018-05-16 17:54:40 +03:00
Dmitry Shulyak 5aae87aba8
Prevent sending messages to flaky peers (#917) 2018-05-15 20:08:31 +03:00
Adrià Cidre 08b4d515c6
[#ISSUE-937] : Move mail server implementation to status-go repo (#942)
* [#ISSUE-937] : Move mail server implementation to status-go repo

* Applying dep-ensure
2018-05-11 21:43:07 +02:00
Dmitry Shulyak 7aa508765e Set timesource from config (#915) 2018-05-08 13:38:54 +03:00
Dmitry Shulyak 84cb5ca917 Ensure that wg.Add is called before wg.Wait
Now if Add is to be called it will be called before Wait, this is achieved
by doing following:
- if cancel() gets lock first and closes channelCh before spawnSync is called
we will exit right away
- if not than we will ensure that we hold a lock until syncers are spawned
so that cancel() will be blocked for this time and it will prevent whole Terminate() from
progressing
2018-05-04 10:28:15 +02:00
Dmitry Shulyak 707221954f
Make whisper tolerant to local time skews (#864)
This change adds adds an ability to use different source of time for whisper:

when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.

So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.

In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.

Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.

Closes: #687
2018-05-04 11:23:38 +03:00
Pedro Pombeiro acbf251a3c Fix race condition in `LDBDatabase.Close()`
- https://jenkins.status.im/job/status-go/job/race-check/54/consoleFull
2018-05-03 20:39:42 +03:00
Frank Mueller 123382cdf1
LevelDB vendor has now the same version as used in go-ethereum 1.8.5 (#883)
* LevelDB version is now same as in go-ethereum

* make dep-ensure untracked LevelDB storage.go
2018-05-01 19:09:19 +02:00
Pedro Pombeiro 7074048fa5 Add patch to fix race condition in `Downloader.Cancel`
- Submitted patch upstream in https://github.com/ethereum/go-ethereum/pull/16585
- Fixes Jenkins build in https://jenkins.status.im/job/status-go/job/race-check/48/
2018-05-01 16:29:27 +02:00
Frank Mueller 30cf19e0ae
Wrong command order (#879) 2018-04-30 23:36:28 +02:00
Igor Mandrigin f4cd8d27b5 Add `shhext_getNewFilterMessages` function to RPC APIs.
This function returns only the new messages from the filter, never
returns the same message for the same user twice.
2018-04-27 15:24:37 +02:00
Pedro Pombeiro a23b607597 Fix race condition in `0016-fix-leveldb-issue.patch`
https://jenkins.status.im/job/status-go/job/race-check/45/
2018-04-26 17:19:32 +02:00
Dmitry Shulyak b37fda7731
Update go-ethereum to 1.8.5 (#854)
* Rebase on 1.8.5

* Remove outdated patches and apply all others

* Use shh_post that returns hash

* Use bloom filter for request to mailserver

* Remove tests for sending messages without subbing first

* Fix deadlock in ethdb

* Expect null if receipt is not yet created

* Subscribe to messages before sending them in whisper test
2018-04-24 18:50:26 +03:00
Dmitry Shulyak 2f2dfe16c0
Add default peer limits configuration (#830)
* Add default peer limits configuration

If discovery is enabled for a given cluster - we will set a default
expected number of peers for each enabled service. For example:

- if cluster is rinkeby has a discovery enabled we will
  check which services are enabled
- if whisper is enabled we will set min and max limits by default
- if les is enabled and infura is not used we will set limits too

When statusd is used - configuration must be provided using configuration
supported by statusd.


* Fix deadlock in les peer set
2018-04-20 15:23:18 +03:00
Igor Mandrigin 4cc6028d59 Implement `personal_sign`.
This commit implements `personal_sign` RPC or web3 personal.sign
methods.

NB! Contains breaking API changes.
2018-04-17 11:18:58 +02:00
Adam Babik 0d652c3851
Expose only public APIs in inproc RPC client [breaking-change] (#815)
`CallRPC` binding, which is used as a provider for web3.js, exposes only public or whitelisted APIs.
2018-04-12 18:17:10 +02:00
Dmitry Shulyak ba9a25e284
Implement shh api extension that allows to confirm that message is sent (#814)
* Implement shh api extension that allows to confirm that message is sent

* Add a patch

* Fix linter

* Add readme

* Add tests for tracker

* Address review
2018-04-11 18:41:51 +03:00
Dmitry Shulyak b71b7eeadb Use web3code from separate repo and remove static 2018-04-09 15:42:36 +02:00
Dmitry Shulyak 38d7194a2a Use chain aware signer in simulated backend 2018-04-09 14:41:50 +03:00
Dmitry Shulyak 3f2e39d525 Update vendor because new geth modules were imported 2018-04-09 14:41:50 +03:00
Dmitry Shulyak 74ce515ab2
Add metrics to discv5 (#749) 2018-04-05 16:48:59 +03:00
Evgeny Danienko e2e2af73d7
isForwarded constants introduced 2018-03-27 18:22:29 +03:00
Evgeny Danienko b4c1cd7212
send in-house messages without subscription 2018-03-27 17:47:15 +03:00
Evgeny Danienko de0c03ca4e
patch sending message without subscription applied 2018-03-26 19:30:31 +03:00
Igor Mandrigin cfb3e6a080
Fix datarace when stopping a node. (#723) 2018-03-05 21:36:32 +01:00
Pedro Pombeiro 6cdea4ef97
Update codebase to leverage Whisper v6 (#703)
* Update project to use Whisper v6. Part of #638

* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.

* Enable light mode on Whisper v6 for non-mail servers. Part of #638

* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)

* Update bootnode addresses in staticnodes.json. Part of #638

* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638

* Move MakeTestNodeConfig to utils. Part of #638

* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
2018-03-02 10:25:30 +01:00
Dmitry Shulyak 988cd73217
Use newer CHT defined in go-ethereum (#711) 2018-03-01 15:23:41 +02:00
Pedro Pombeiro e4cbce12c4
Update to go-ethereum 1.8.1 (#702)
* Update `github.com/ethereum/go-ethereum` package to 1.8.1 branch. Part of #638
* Fix code due to some signature changes. Part of #638
* use upstream for whisper backend
* Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #638
* Take into account the DNS rebinding protection introduced in 1.8.0 by adding exception for localhost. Part of #638
* Add patches required for cross-compiled builds starting with geth 1.8.0. Only applied during build. Part of #638
* Update expected JSON result in `TestRegressionGetTransactionReceipt()` and `TestCallRawResultGetTransactionReceipt()`. Part of #665
* Fix some failing e2e tests. Part of #638
* Address comments in PR #702. Part of #638
2018-02-27 11:39:30 +01:00
Igor Mandrigin abb5df88d1
Wait for `Rollback` before closing the DB in the light client. (#696)
* Wait for `Rollback` before closing the DB in the light client.

* Unsubscribe from mux events properly, get rid of `time.Sleep`.
2018-02-23 11:49:57 +01:00
Igor Mandrigin c06d58addd Remove `notifications` package from go-ethereum. 2018-02-22 11:10:21 +01:00
Adam Babik 04d216dcc5 Remove non-go option (#682) 2018-02-21 15:37:46 +01:00
Dmitry Shulyak b2580c79d7
Add destructive tests (#646)
Network disconnect is introduced by removing default gateway, easily reversible condition.

On my local machine it takes 30 seconds for peers to reconnect after connectivity is restored. As you guess this is not an accident, and there is 30 seconds timeout for dial expiration. This dial expiration is used in p2p.Server to guarantee that peers are not dialed too often.

Additionally I added small script to Makefile to run such tests in docker environment, usage example:

```
make docker-test ARGS="./t/destructive/ -v -network=4"
```
2018-02-19 17:32:58 +02:00
Alex Kohler 87b12ddced Remove DefaultMinimumPow patches in favor of using whisper config (#683) 2018-02-19 15:53:40 +01:00
Dmitry Shulyak 4a49e840e3 Remove status backend from les internal api (#670) 2018-02-16 15:40:37 +01:00
Igor Mandrigin 004d3ef249 Remove unused `0007-README.patch`. 2018-02-15 13:08:32 +01:00
Igor Mandrigin 2e03cbcb5d Remove outdated `0003-dockerfiles-wnode-swarm.patch`. 2018-02-15 12:51:37 +01:00
Caner Çıdam 15c72f3c6b Apply geth patches on the fly #653 (#660) 2018-02-14 18:44:27 +01:00
Pedro Pombeiro 857b72e9fd
Create whisperv6 patch versions of `0004-whisper-notifications.patch` and `0009-whisper-envelopes-tracing.patch`. Closes #637 2018-02-12 16:44:04 +01:00
Pedro Pombeiro 9976018978
Create temporary whisperv6 patch file. Closes #636 2018-02-12 16:36:55 +01:00
Igor Mandrigin 9cc9982a6a
[#639] Disable filter removal in `go-ethereum`. (#650) 2018-02-12 13:02:25 +01:00
Dmitry Shulyak d0ef64a177 Maintain local copy of the nonce for each used address (#538) 2018-02-08 00:23:57 +02:00
Igor Mandrigin 156bbe1986 [#429] fix npe in geth's filter system. 2018-02-02 15:10:30 +02:00
Adam Babik 89d89681a3
Add support for metrics (#558)
It adds support for metrics (expvar and Prometheus) along with docker-compose files to run a Whisper test cluster.
2018-01-30 12:51:48 +01:00
Igor Mandrigin fb3d2ff6fe
Introduce the dependency vendoring tool: `dep`. (#551)
* Introduce `dep`, the dependency vendoring tool.

Use commits from `go-ethereum@release/1.7` for most of the dependencies.

* Update dependencies.
2018-01-25 14:08:43 +01:00
Adam Babik c153a60dc3
Clean up whisper log delivery (#555) 2018-01-19 15:53:16 +01:00
Dmitry Shulyak 0771e7d1b7 Use single codepath for sending transactions to a local and remote nodes (#527)
- new EthereumTransactor that provides higher level API for working with ethereum network, and it is fully conformant with ethclient
- new test rpc service that improves flexibility and coverage of txqueue manager tests
- run complete transaction sequantially for each address
- go-ethereum: Get pending nonce from transaction pool
- add a patch with getting nonce from transaction pool
2018-01-18 17:55:17 +01:00
Ivan Daniluk f93cd81d83
Upgrade to geth 1.7.3 and add geth patches (#492) 2018-01-15 21:26:41 +01:00
Gene 8acfc71043 Remove unused vendor libraries #198 (#480) 2017-11-30 13:36:20 +01:00
Ivan Daniluk 4cd362213f
Add env var to tune header writes delay (#464) 2017-11-22 21:13:11 +02:00
Ewetumo Alexander cb5ccb52c4 Emit messages logs for processing and sorting out messaging problems (#420)
* Add DeliveryService to Whisper to track incoming and outgoing messages.
* Have log tagged log lines for incoming and outgoing messages.
2017-11-06 14:10:52 -05:00
Adam Babik a19e502686 Remove go.rice in favour of go generate (#444)
As fetch.go was the only consumer of go.rice, it could be safely removed in favour of go-bindata and go generate.
Motivation to do that was the fact that go.rice throws panic in iOS when compiled with gomobile.
2017-11-03 18:11:37 -04:00
Adam Babik b9372459cc Rebase on geth 1.7.2 (#402) 2017-10-17 12:58:04 +03:00
Adam Babik 90acfedf7a Rebase geth 1.7.0 (#353) 2017-10-10 12:38:49 +03:00
Adam Babik 750612f2bc fix SendTransaction signature in go-ethereum (#355)
I created a separate method SendTransactionWithPassphrase which accepts passphrase as a second argument. It's an exact copy of SendTransaction except for calling wallet.SignTxWithPassphrase.
2017-09-22 18:16:22 +03:00
Oskar Thorén f88de68f86 Push notifications: ability to trigger notifications via public API (#327)
This PR provides a way for status-react to trigger push notifications to contacts whose FCMToken they possess. It thus solves the basic user story as outlined in #326
2017-09-15 18:57:34 +03:00
Ivan Daniluk 6a096607cf Add FetchAPI support and fix loop race [upd] #289 (#293)
This PR adds Fetch API and fixes #289 by using concurrency safe Otto VM wrapper wherever it's possible. This involves new package geth/jail/vm that is used by jail and by our forked ottoext/{fetch/timers/loop} packages.

It also adds more tests that are supposed to be run with --race flag of go test.
2017-09-08 14:55:17 +03:00
Adam Babik 39aeb3b09d vendor: clean up go-ethereum (#299) 2017-09-07 10:35:10 +02:00
Adam Babik 8fb2424ea5 Port the transactions queue from go-ethereum to status-go (#261)
Now all transactions in both cases are waiting for CompleteTransaction or DiscardTransaction to proceed independently from their destination: upstream of local
2017-09-04 15:56:58 +03:00
Oskar Thorén 680090bdf9 vendor: Upgrade go-stack 2017-09-03 11:15:03 +02:00
Ivan Daniluk 75398a19d5 Fix "Slow CreateAccount" #251 (#283) 2017-09-02 03:17:34 +07:00
Adam Babik 7433828a26 Validate JSON config (#240)
* exposed ValidateNodeConfig()
* extended NodeConfig interface with Validate() method
* added validate tag to some NodeConfig fields
2017-08-10 22:31:29 +07:00
Ivan Tomilov ebd77aabe2 Merging bug/whisper-on-geth1.6.1 (#236) which acts like develop
* static: updates Whisper test (to work with Geth 1.6.1)
* jail: VM persistence implemented
* jail: sendMessage/showSuggestions minor fixes (to be squashed)
* node: CHT and boot nodes auto-load implemented
* Replaced CHT data file from farazdagi's to tiabc's
* Rewrote config_test.go using testify having reduced it twice in size
* Increased SyncTime and panic timeout in tests
* Fixed test - remove go default test to testify/suite (#207)
* Add flag setup for RPCEnabled and add comment (#225)
* jail: register method handlers before running initial js in jail (#226)
* Console Jail Mod #179 (#228)
* Added ./statusd-data into .gitignore
* Increased log level for the test node from INFO to ERROR
* Add call to loop.Run to evaluate all setTimeout/setIntervals methods. (#208)
* Rebase onto geth1.6.7 (#232)
* Got back sync duration from 60s to 30s, updated bindata.go
2017-08-04 23:14:17 +07:00
Ewetumo Alexander b50c46caa8 Add SetTimeout/SetInterval to JailCell otto.Otto, closes #189
* Commit initial change for settimeout/setinterval

* Add initial tests for jail setTimeout/setInterval

* Add ottoext dependency

* Add fetch jail test with function

* Add dependencies of fetch from ottoext

* Refactor with regards to PR review

* Refactor with regards to PR review

* Fix syntax errors

* Fix missing return statement
2017-07-13 18:04:47 +07:00
Victor Farazdagi 6b3f7aabdf node: expose in-proc RPC (CallRPC), closes #144 2017-05-28 17:05:33 +03:00
Victor Farazdagi 349103de1a major refactoring: node manager, tx queue, accounts, jail
- node: signals and node reset, fixes #152
- tests update (testify is used)
- node manager refactored, race conditions fixed
- node wrapper has been removed, we rely on go-ethereum node now
- tx queue refactored
- jail refactored
- backend and API introduced
2017-05-24 00:29:20 +03:00
Victor Farazdagi 9c679d6bb7 verndor: stretchr/testify added 2017-05-23 06:04:03 +03:00
Victor Farazdagi c7ef35d414 node: allow CHT config per sub-cluster 2017-05-17 19:26:07 +03:00
Victor Farazdagi 82fe888853 whisper5: minor fixes 2017-05-16 11:02:34 +03:00
Andreas Schmid 01d3aa9fe1 whisperv5: fix protocol of whisperv5 envelopes 2017-05-16 10:51:11 +03:00
Victor Farazdagi 4f27a79d83 config: homestead and rinkeby confg + allow dev/prod modes 2017-05-16 10:51:11 +03:00
Victor Farazdagi 2f0c93fd3b refactor: linters + recommendations applied, closes #146 2017-05-15 08:19:46 +03:00
Victor Farazdagi 311f2b80c5 vendor: CHT update (block 925696, Revival Ropsten) 2017-05-14 20:25:20 +03:00
Victor Farazdagi 92afd0d47e vendor/whisper, statusd: push notifications implemented, closes #135 2017-05-03 16:09:48 +03:00
Victor Farazdagi b7c38238bd rebase: new packages added to vendors
Packages added:
- golang.org/x/net/context updated
- mattn/go-colorable
- karalabe/hid
- mattn/go-color
- go-stack/stack
2017-05-03 16:00:35 +03:00
Victor Farazdagi 44d4335a7d vendors: geth updated to 1.6.1-unstable-without-whisper 2017-05-03 16:00:35 +03:00
Victor Farazdagi 980d054a06 vendor: CHT update (block 761856, Revival Ropsten) 2017-04-19 08:33:28 +03:00
Victor Farazdagi ef0148d76a whisper: fix issue with PoW calculation 2017-04-03 14:57:51 +03:00
Victor Farazdagi 3ce913f661 geth, jail: Whisper ver.5 enabled, tests expanded 2017-04-03 14:57:51 +03:00
Victor Farazdagi 27700f5763 vendor: CHT update (block 655360, Revival Ropsten) 2017-03-30 16:48:04 +03:00
Victor Farazdagi ccb9c4f6d8 vendor: CHT update (block 725183) 2017-03-29 00:49:27 +03:00
Victor Farazdagi 21a891d74c vendor: CHT update (block 659647) 2017-03-10 04:43:21 +03:00
Victor Farazdagi bea12ee9fe vendor: adds karalabe/gousb/usb 2017-02-27 16:22:45 +03:00
Victor Farazdagi f3cd191a06 vendor: 1.5.9-stable (non-broken LES) 2017-02-27 15:40:42 +03:00
Victor Farazdagi 158db66500 vendor: 1.5.8-stable (non-broken LES) 2017-02-23 20:24:40 +03:00
Victor Farazdagi b2bbf1353c vendor: 1.5.7-stable (non-broken LES) 2017-02-23 05:18:01 +03:00
Victor Farazdagi 4284b7743e vendor: 1.5.6-stable (non-broken LES) 2017-02-23 05:12:38 +03:00
Victor Farazdagi 87e345c1cc fixes issue when non-originator can complete a tx 2017-02-01 20:08:03 +03:00
Victor Farazdagi 09d021e598 update vendors 2016-12-07 23:54:59 +03:00
Victor Farazdagi 8855cad7be Integrate with geth 1.5.3 (1.5.4-unstable). Fixes #67 2016-11-26 12:43:31 +03:00
Victor Farazdagi b6e356a352 fixes issue with wrong tx signer chainId 2016-11-25 11:05:03 +03:00
Victor Farazdagi 6d061f4249 updates vendors (geth 1.5.3-unstable) 2016-11-25 08:51:24 +03:00
Victor Farazdagi c29dedf84a implements DiscardTransaction(), required for #32 2016-10-31 02:28:30 +03:00
Victor Farazdagi 6e18cbb5d4 tx send: allow to re-send + error signaling. Fixes #49, #51 2016-10-28 17:33:25 +03:00
Victor Farazdagi 2f3cebd047 Updates go-ethereum to the latest light-stable 2016-10-21 12:03:15 +03:00
Victor Farazdagi 6abbca9935 jail: more robust cell ticket acquire/release 2016-10-12 20:51:25 +03:00
Victor Farazdagi 745c3a46c1 Updates vendors 2016-10-12 01:59:43 +03:00
Victor Farazdagi f11c6421a1 Latest light-stable code pushed to vendors 2016-10-12 01:56:36 +03:00
Victor Farazdagi 009fab27fd Rebased against the latest light-stable 2016-09-29 22:51:33 +03:00
Victor Farazdagi 7505281a63 eth.accounts rely on ext. keys now (account + sub accounts) 2016-09-28 10:38:44 +03:00
Victor Farazdagi edd8763c3c Major code refactoring
- geth package implemented
- jail package implemented (and tested)
- cgo and xgo (android, ios) builds w/o any issues
- better conditional compilation of signals code
2016-09-15 06:13:40 +03:00
Victor Farazdagi 5fb4aef1cc Initial refactoring + vendor update (after rebase) 2016-09-15 06:12:25 +03:00
Jarrad Hope c5f9cbcfc9 successful build 2016-06-18 20:38:34 +02:00
Daniel Whitenack ae676e8140 statusgo executable with binding and flags 2016-06-15 14:50:35 -05:00
Daniel Whitenack e21ca3f08a new account binding, with example usage 2016-06-15 08:54:07 -05:00
Daniel Whitenack 18d53211d9 godeps initial commit, test script 2016-06-14 11:54:23 -05:00