Commit Graph

134 Commits

Author SHA1 Message Date
Adam Babik 4e386f4097
Update version format of artifacts in Jenkins (#1173) 2018-08-29 12:52:19 +02:00
Adam Babik a571eeb7a4
add patch 0036-fix-whisper-expire-loop (#1175) 2018-08-28 10:49:08 +02:00
Dmitry Shulyak f150d678de
Update geth to 1.8.14 (#1171)
* Update to geth v1.8.14

* Remove patches that were merged upstream

* Apply patches before 0016

* Fix 0016 and apply it

* Apply everything else

* Pass gas limit as a second argument to simulated backend
2018-08-27 11:22:21 +03:00
Adam Babik 16c7b94eb5
Add labels to docker images and add Jenkinsfile to build docker images (#1160)
* fixed generating `params.Version`
* added labels to all Docker images including git_commit and the author of the build (using `$USER` env variable)
2018-08-24 12:25:07 +02:00
Dmitry Shulyak 2cfcde0283
Remove patches to discovery v5 (#1144) 2018-08-24 11:24:22 +03:00
Dmitry Shulyak 6858662d59
Proxy records found in ethereum discovery v5 (#1165) 2018-08-24 06:17:32 +03:00
Dmitry Shulyak d8b5ba7fbf
Add generated cht for ropsten and mainnet (#1166) 2018-08-24 06:05:06 +03:00
Dmitry bfdf823628 Add delay to refresh cycle with patch 2018-08-15 04:25:18 +03:00
Dmitry 6c6fb4d0b3 Remove patch 0031 2018-08-15 04:25:18 +03:00
Adam Babik 7c70ce02bc
allow building status-go from tags (#1122) 2018-08-10 10:28:04 +02:00
Adam Babik 4219d56909
Fix xgo patch 0002 (#1141) 2018-08-08 09:39:51 +02:00
Adam Babik ecc39735f5
Upgrade geth to 1.8.13 (#1140) 2018-08-07 15:31:06 +02:00
Jakub Sokołowski ef7da275d4
fix ENTRYPOINT syntax and add CMD for easier use by new people; 2018-07-11 13:45:27 +02:00
Jakub 6d4f657106 Add ENTRYPOINT for status-go image (#1084) 2018-07-10 11:29:40 +02:00
Ivan Daniluk 93210061ad
Add goroutines count to geth metrics (#1077) 2018-07-03 20:15:16 +02:00
Andrea Franz 809db97e54
mailserver pagination (#1039)
* mailserver sends envelopes in descending order

* add limit value in mailserver request payload

* mailserver sends messages up to the limit specified in the request

* update Archive method to return key and error

* processRequest returns the next page cursor

* add cursor to mailserver request

* add limit and cursor to request payload

* fix request limit encoding

* wait for request completed event in TrackerSuite/TestRequestCompleted

* add cursor to mailserver response

* fix cursor position in payload

* add e2e test for mail server pagination

* validate mail server response size

* remove old limitReached var

* fix lint warnings

* add whisper patch

* fix tests after rebase

* check all return values to avoid lint warnings

* check that all messages have been retrieved after 2 paginated requests

* fix lint warnings

* rename geth patch

* merge mailserver patches into one

* add last envelope hash to mailserver response and EventEnvelopeAvailable event

* update whisper patch

* add docs to MailServerResponse

* update whisper patch

* fix tests and lint warnings

* send mailserver response data on EventMailServerRequestCompleted signal

* update tracker tests

* optimise pagination test waiting for mailserver to archive only before requesting

* rollback mailserver interface changes

* refactoring and docs changes

* fix payload size check to determine if a limit is specified

* add more docs to the processRequest method

* add constants for request payload field lengths

* add const noLimits to specify that limit=0 means no limits
2018-07-02 09:38:10 +02:00
Andrea Franz b913c98573
remove formatting from metrics.ResettingTimer when requested in raw format (#1052) 2018-06-27 11:46:50 +02:00
Pedro Pombeiro 3b052098fe Update to geth 1.8.11. Closes #1033
- Replace deprecated common.Hex with hexutil.Encode.
- Remove upstreamed 0010-geth-17-fix-npe-in-filter-system.patch.
- Remove upstreamed 0020-discv5-metrics.patch.
- Remove upstreamed 0026-ethdb-error-deadlock.patch.
- Update goleveldb to same version used by geth 1.8.11.
- Update PublicTransactionPoolAPI.GasPrice return type to match type in internal geth interface.
2018-06-22 18:04:13 +02:00
Adrià Cidre 51ab9053ab
[#1040] Patch go-ethereum reducing from Warn to Info "message loop" logs (#1048) 2018-06-22 11:57:52 +02:00
Andrea Franz fa390a52ae
RequestHistoricMessages response (#1009)
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper

* send p2pRequestResponseCode from mailserver

* send p2p message response to after sending all historic messages

* mailserver sends `whisper.NewSentMessage` as response

* add mailserver Client and p2pRequestAckCode watchers

* send event with envelopeFeed when p2pRequestAckCode is received

* test request completed event in tracker

* rename mailserver response events and code to RequestCompleteCode

* wait for mailserver response in e2e test

* use SendHistoricMessageResponse method name for mailserver response

* fix lint warnings

* add mailserver request expiration

* send mailserver response without envelope

* add `ttl` to Request struct in shhext_requestMessages

* test that tracker calls handler.MailServerRequestExpired

* add geth patch

* rename TTL to Timeout

* split tracker.handleEvent in multiple methods
2018-06-15 17:12:31 +02:00
Pedro Pombeiro 354e23aaf5 Fix issues reported by lint. Part of #1017 2018-06-14 13:52:51 +02:00
Adam Babik 8ce6efc749
Make statusd distinct from StatusIM client (#1008) 2018-06-08 11:20:30 +02:00
Andrea Franz 5d0120ee28
add Dockerfile and Makefile tasks to build statusd-prune docker image (#1005)
* add Dockerfile and Makefile tasks to build statusd-prune docker image

* add image ENTRYPOINT

* set default CMD to statusd-prune
2018-06-07 12:30:18 +02:00
Adam Babik b5f05b1ab5
Fix flaky TestSingleTopicDiscoveryWithFailover and others (#992)
Other changes:
* needed to patch that loop implementation in Discover V5 implementation in go-ethereum,
* fixed TestStatusNodeReconnectStaticPeers,
* fixed TestBackendAccountsConcurrently.
2018-05-26 09:37:13 +02:00
Dmitry Shulyak ed1cdf0418
Custom status bootnodes (#968)
This change makes invalidation mechanism more aggressive. With a primary goal to invalidate short living nodes faster. In current setup any node that became known in terms of discovery will stay in this state until it will fail to respond to 5 queries. Removing them earlier from a table allows to reduce latency for finding required nodes.

The second change, one adds a version for discovery, separates status dht from ethereum dht.
After we rolled out discovery it became obvious that our boot nodes became spammed with irrelevant nodes. And this made discovery process very long, for example with separate dht discovery takes ~2s, with mutual dht - it can take 1m-10m and there is still no guarantee to find a max amount of peers, cause status nodes is a very small part of whole ethereum infra.

In my understanding, we don't need to be a part of ethereum dht, and lower latency is way more important for us.

Closes: #941
Partially closes: #960 (960 requires futher investigations on devices)
2018-05-18 16:43:07 +03:00
Adam Babik 6da469140f
Refactor in-app metrics (#963)
Remove Prometheus and other metric types and use metrics package from go-ethereum.
2018-05-16 17:36:59 +02:00
Dmitry Shulyak ca962ddaa3 Whisper broadcast loop spams way too hard with debug level 2018-05-16 17:54:40 +03:00
Dmitry Shulyak 5aae87aba8
Prevent sending messages to flaky peers (#917) 2018-05-15 20:08:31 +03:00
Ivan Danyliuk 6fec9fa653 Bump up Go version to 1.10 2018-05-09 16:13:34 +02:00
Dmitry Shulyak 7aa508765e Set timesource from config (#915) 2018-05-08 13:38:54 +03:00
Dmitry Shulyak 84cb5ca917 Ensure that wg.Add is called before wg.Wait
Now if Add is to be called it will be called before Wait, this is achieved
by doing following:
- if cancel() gets lock first and closes channelCh before spawnSync is called
we will exit right away
- if not than we will ensure that we hold a lock until syncers are spawned
so that cancel() will be blocked for this time and it will prevent whole Terminate() from
progressing
2018-05-04 10:28:15 +02:00
Dmitry Shulyak 707221954f
Make whisper tolerant to local time skews (#864)
This change adds adds an ability to use different source of time for whisper:

when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.

So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.

In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.

Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.

Closes: #687
2018-05-04 11:23:38 +03:00
Pedro Pombeiro acbf251a3c Fix race condition in `LDBDatabase.Close()`
- https://jenkins.status.im/job/status-go/job/race-check/54/consoleFull
2018-05-03 20:39:42 +03:00
Dmitry Shulyak c8a553f9c1 Fix 0002 xgo patch which changed because metrics got forked 2018-05-02 22:59:54 +02:00
Pedro Pombeiro 4438d116b1 Ensure that Travis CI fails the build if the patches are incorrect 2018-05-02 13:28:39 +02:00
Dmitry Shulyak 7f81323fdc Remove download patch 0027 2018-05-02 12:04:17 +02:00
Pedro Pombeiro 7074048fa5 Add patch to fix race condition in `Downloader.Cancel`
- Submitted patch upstream in https://github.com/ethereum/go-ethereum/pull/16585
- Fixes Jenkins build in https://jenkins.status.im/job/status-go/job/race-check/48/
2018-05-01 16:29:27 +02:00
Igor Mandrigin f4cd8d27b5 Add `shhext_getNewFilterMessages` function to RPC APIs.
This function returns only the new messages from the filter, never
returns the same message for the same user twice.
2018-04-27 15:24:37 +02:00
Pedro Pombeiro a23b607597 Fix race condition in `0016-fix-leveldb-issue.patch`
https://jenkins.status.im/job/status-go/job/race-check/45/
2018-04-26 17:19:32 +02:00
Dmitry Shulyak b37fda7731
Update go-ethereum to 1.8.5 (#854)
* Rebase on 1.8.5

* Remove outdated patches and apply all others

* Use shh_post that returns hash

* Use bloom filter for request to mailserver

* Remove tests for sending messages without subbing first

* Fix deadlock in ethdb

* Expect null if receipt is not yet created

* Subscribe to messages before sending them in whisper test
2018-04-24 18:50:26 +03:00
Dmitry Shulyak 2f2dfe16c0
Add default peer limits configuration (#830)
* Add default peer limits configuration

If discovery is enabled for a given cluster - we will set a default
expected number of peers for each enabled service. For example:

- if cluster is rinkeby has a discovery enabled we will
  check which services are enabled
- if whisper is enabled we will set min and max limits by default
- if les is enabled and infura is not used we will set limits too

When statusd is used - configuration must be provided using configuration
supported by statusd.


* Fix deadlock in les peer set
2018-04-20 15:23:18 +03:00
Igor Mandrigin 4cc6028d59 Implement `personal_sign`.
This commit implements `personal_sign` RPC or web3 personal.sign
methods.

NB! Contains breaking API changes.
2018-04-17 11:18:58 +02:00
Adam Babik 0d652c3851
Expose only public APIs in inproc RPC client [breaking-change] (#815)
`CallRPC` binding, which is used as a provider for web3.js, exposes only public or whitelisted APIs.
2018-04-12 18:17:10 +02:00
Dmitry Shulyak ba9a25e284
Implement shh api extension that allows to confirm that message is sent (#814)
* Implement shh api extension that allows to confirm that message is sent

* Add a patch

* Fix linter

* Add readme

* Add tests for tracker

* Address review
2018-04-11 18:41:51 +03:00
Dmitry Shulyak b71b7eeadb Use web3code from separate repo and remove static 2018-04-09 15:42:36 +02:00
Dmitry Shulyak 38d7194a2a Use chain aware signer in simulated backend 2018-04-09 14:41:50 +03:00
Dmitry Shulyak 74ce515ab2
Add metrics to discv5 (#749) 2018-04-05 16:48:59 +03:00
Adam Babik e9c2292776
Switch mockgen to reflect mode (#800)
Switch mockgen to use reflect mode as source mode does not support vendored packages.
2018-04-05 15:14:47 +02:00
Dmitry Shulyak 5e5e497822 Initial version of peer docs 2018-04-04 13:27:35 +03:00
Evgeny Danienko e2e2af73d7
isForwarded constants introduced 2018-03-27 18:22:29 +03:00
Evgeny Danienko b4c1cd7212
send in-house messages without subscription 2018-03-27 17:47:15 +03:00
Evgeny Danienko aed16a9071
patch 2018-03-26 18:17:17 +03:00
Adam Babik 8b94ac00bb
Update mainnet dev cluster config (#745) 2018-03-22 15:58:28 +01:00
Frank Mueller dc4a0a98ed Fix setting of bootstrap nodes (#744)
Fix dropped bootstrap nodes setting and rename of cluster config and separation of nodes
2018-03-20 16:05:21 +02:00
Pedro Pombeiro d84510de92
Remove unused JavaScript tests for RPC API (#737) 2018-03-16 18:49:42 +01:00
Frank Mueller 81c8669518
Code review for Mainnet (#706)
* Add debug log to test package t

* Add comments for test restrictions

* Add explicit errors for mainnet

* Panic instead of error in case of mainnet during tests

* Now panic with illegal network

* Validate the compiler flag for mainnet

* Made test for unset mainnet explicit

* Forgot to use my flag

* Linter has troubles with unused field

* Add default network id

In e2e tests there's no default anymore in go code. So ensure in Makefile.

* Fix dumb syntax error

* CI needs build with Mainnet

* Explicit rebuild only for tests

* No more separation of mainnet and testnet builds

* Build flags now set in Makefile
2018-03-15 12:36:28 +01:00
Igor Mandrigin cfb3e6a080
Fix datarace when stopping a node. (#723) 2018-03-05 21:36:32 +01:00
Pedro Pombeiro 6cdea4ef97
Update codebase to leverage Whisper v6 (#703)
* Update project to use Whisper v6. Part of #638

* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.

* Enable light mode on Whisper v6 for non-mail servers. Part of #638

* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)

* Update bootnode addresses in staticnodes.json. Part of #638

* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638

* Move MakeTestNodeConfig to utils. Part of #638

* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
2018-03-02 10:25:30 +01:00
Dmitry Shulyak 988cd73217
Use newer CHT defined in go-ethereum (#711) 2018-03-01 15:23:41 +02:00
Pedro Pombeiro e4cbce12c4
Update to go-ethereum 1.8.1 (#702)
* Update `github.com/ethereum/go-ethereum` package to 1.8.1 branch. Part of #638
* Fix code due to some signature changes. Part of #638
* use upstream for whisper backend
* Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #638
* Take into account the DNS rebinding protection introduced in 1.8.0 by adding exception for localhost. Part of #638
* Add patches required for cross-compiled builds starting with geth 1.8.0. Only applied during build. Part of #638
* Update expected JSON result in `TestRegressionGetTransactionReceipt()` and `TestCallRawResultGetTransactionReceipt()`. Part of #665
* Fix some failing e2e tests. Part of #638
* Address comments in PR #702. Part of #638
2018-02-27 11:39:30 +01:00
Igor Mandrigin abb5df88d1
Wait for `Rollback` before closing the DB in the light client. (#696)
* Wait for `Rollback` before closing the DB in the light client.

* Unsubscribe from mux events properly, get rid of `time.Sleep`.
2018-02-23 11:49:57 +01:00
Caner Çıdam 96dc15e167 fix exit code check (#697) 2018-02-22 23:09:09 +01:00
Igor Mandrigin c06d58addd Remove `notifications` package from go-ethereum. 2018-02-22 11:10:21 +01:00
Pedro Pombeiro d02c2f16d8
Fix `-log` option in `wnode-test-cluster/docker-compose.yml`
- Was causing `wnode_1     | flag provided but not defined: -log DEBUG` error
2018-02-21 17:57:24 +01:00
Alex Kohler c5b9f00467 Adding update-geth target (#673) 2018-02-20 19:04:25 +01:00
Alex Kohler 87b12ddced Remove DefaultMinimumPow patches in favor of using whisper config (#683) 2018-02-19 15:53:40 +01:00
Pedro Pombeiro 7410532879 Patcher improvement (#681)
Improve patcher tool:
* fix broken `-p` flag,
* rename it to `-b`,
* add a new `-p` flag that allows specifying the source of the patches, so that it can be used for more than just go-ethereum patches,
* make it work correctly independently of the order that options are specified in the command line.
2018-02-18 09:14:27 -07:00
Dmitry Shulyak 4a49e840e3 Remove status backend from les internal api (#670) 2018-02-16 15:40:37 +01:00
Alex Kohler a2431b3e2b Fix typo in README 2018-02-16 03:45:06 +01:00
Igor Mandrigin 004d3ef249 Remove unused `0007-README.patch`. 2018-02-15 13:08:32 +01:00
Igor Mandrigin 2e03cbcb5d Remove outdated `0003-dockerfiles-wnode-swarm.patch`. 2018-02-15 12:51:37 +01:00
Igor Mandrigin 18373da969 Remove mentions of our fork of `go-ethereum` from docs. 2018-02-15 11:16:59 +01:00
Caner Çıdam 15c72f3c6b Apply geth patches on the fly #653 (#660) 2018-02-14 18:44:27 +01:00
Adam Babik 24aa1b0f1c add node_peers Prometheus metric 2018-02-13 15:18:29 +01:00
Adam Babik 404f633f2b add doc string to get-docker-image-tag.sh 2018-02-13 10:30:47 +01:00
Adam Babik 9ed2663b6e add docker-image-tag target to Makefile 2018-02-13 10:30:47 +01:00
Pedro Pombeiro ffef611295
Tear `0013-whisperv6-notifications.patch` into two different patches (#657) 2018-02-13 10:14:57 +01:00
Dmitry Shulyak 5503278899
Fix wnode cluster after recent changes (#654) 2018-02-12 18:57:19 +02:00
Pedro Pombeiro 857b72e9fd
Create whisperv6 patch versions of `0004-whisper-notifications.patch` and `0009-whisper-envelopes-tracing.patch`. Closes #637 2018-02-12 16:44:04 +01:00
Pedro Pombeiro 9976018978
Create temporary whisperv6 patch file. Closes #636 2018-02-12 16:36:55 +01:00
Pedro Pombeiro f5393274ff
Fix a couple of patches where new files are created so that new files are marked as such
Signed-off-by: Pedro Pombeiro <pombeirp@users.noreply.github.com>
2018-02-12 15:56:03 +01:00
Pedro Pombeiro 51f1434a77
Make patcher apply the patches from the folder relative to the script itself 2018-02-12 15:55:32 +01:00
Igor Mandrigin 9cc9982a6a
[#639] Disable filter removal in `go-ethereum`. (#650) 2018-02-12 13:02:25 +01:00
Caner Çıdam a4c36fe96a Move non-go related files to the `_assets` folder. #621 (#630) 2018-02-09 14:38:18 +01:00