MailServer pruning was implemented as a separate command but that required stopping a mail server and executing the command manually. This change builds pruning into MailServer and can be set using MailServerDataRetention in WhisperConfig.
These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history
When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.
If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.
If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.
We will have two components that will rely on this logic:
1. requesting history
If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.
2. confirmation from selected mail server
Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.
I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server
Currently this feature is hidden, as certain changes will be necessary in status-react.
partially implements: https://github.com/status-im/status-go/issues/1285
- Replace command line flags with `-c` config flag. Part of #1180
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
Remove `PendingSignRequests` queue from the sign module.
This closes#1027 by removing the pending sign requests queue dependency from the SendTransaction, SignMessage and Recover.
Update vendor
Integrate rendezvous into status node
Add a test with failover using rendezvous
Use multiple servers in client
Use discovery V5 by default and test that node can be started with rendezvous discovet
Fix linter
Update rendezvous client to one with instrumented stream
Address feedback
Fix test with updated topic limits
Apply several suggestions
Change log to debug for request errors because we continue execution
Remove web3js after rebase
Update rendezvous package
This change makes invalidation mechanism more aggressive. With a primary goal to invalidate short living nodes faster. In current setup any node that became known in terms of discovery will stay in this state until it will fail to respond to 5 queries. Removing them earlier from a table allows to reduce latency for finding required nodes.
The second change, one adds a version for discovery, separates status dht from ethereum dht.
After we rolled out discovery it became obvious that our boot nodes became spammed with irrelevant nodes. And this made discovery process very long, for example with separate dht discovery takes ~2s, with mutual dht - it can take 1m-10m and there is still no guarantee to find a max amount of peers, cause status nodes is a very small part of whole ethereum infra.
In my understanding, we don't need to be a part of ethereum dht, and lower latency is way more important for us.
Closes: #941
Partially closes: #960 (960 requires futher investigations on devices)
add mailserver cleaner
use memstorage for leveldb in tests
avoid write if batch size is 0
add comments
add cmd/statusd-prune
rmeove batch size var in prune method
validate range values
pass only flag name to missingFlag
refactor Cleaner.prune method
update batch not to be a pointer
removed extra batch counter increment
don't increment counter if batch returns errors
add README
* Make it possible to explicitly disable discovery
Discovery will be disabled in following cases:
- if there are not bootnodes - v5 server will be disabled
because there is no point in running it
- if user defined in config NoDiscovery=true this value will be preserved
even if we have bootnodes
So, basically discovery will be always enabled by default on mobile, unless
it is explicitly specified otherwise.
When statusd is used current behavior is that discovery is disabled by default.
I kept it in this change, but it would be better to change it.
* Fix leftovers
* Add wait group to peer pool to protect from races with p2p.Server
* Change fields only when all goroutines finished
* Turn off discovery after topic searches are stopped
* Don't set period to nil to avoid race with SearchTopic
* Close period chan only when all writers are finished
* add `-status` flag to enable the Status service
* remove status from default APIModules and add it only from statusd if specified
* remove AddAPIModule method
* allow -status flag values to be http or ipc
* add LogEnabled attribute to NodeConfig, used in the call from status-react
* fix use of OverrideRootLogin in tests
* enable logger in tests based on LogLevel
* move LogEnabled before LogFile
This change will greatly simplify writing unit tests when a node is required but data persistence is irrelevant.
I also Introduced some refactoring and unit tests for `StatusNode`.
- [x] [#797] : Remove unused methods PopulateStaticPeers, ReconnectStaticPeers, removeStaticPeers, removePeer
- [x] [#797] : Rename node.Manager to node. StatusNode and simplify its public api
- [x] [#797] : Rename all references to nodeManager to statusNode
* Rename bootnode to static peers
Also add some groupings between types.
* Remove not needed genesisHash
* More cleanup of bootnodes with static peers
* Add option of cluster configuration file
* New generated bindata.go
* Changes after npm install
* Add argument for cluster configuration file
* Add test for dynamic cluster loading
Not yet sure with name "cluster config".
* Solved conflicts
* Renaming of static peers
* Remove static peers population
* Missing argument for config
* Renaming of static peers to boot nodes for consistency
* Fix of name change
* Cluster config is now cluster data
* Load static nodes from configuration
* Final renaming of var for file content
* Update project to use Whisper v6. Part of #638
* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.
* Enable light mode on Whisper v6 for non-mail servers. Part of #638
* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)
* Update bootnode addresses in staticnodes.json. Part of #638
* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638
* Move MakeTestNodeConfig to utils. Part of #638
* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
Summary:
Filter out gas linter error checks for fmt.Fprintf commands. This required defining a custom linter around gas that additionally included the offending code.
Notes:
Gas format, without piping it through gometalinter, gives output like this:
$ gas -fmt=csv geth/jail/console/console.go
geth/jail/console/console.go,21,Errors unhandled.,LOW,HIGH,"fmt.Fprintf(w, ""%s: %s"", consoleEventName, formatForConsole(fn.ArgumentList))"
Gometalinter, by default, does not grab the line of code when it filters gas errors. To resolve this, I created a wrapper around gas (I wasn't sure what to call this "gas wrapper", I opted for gasv2, open to other names).
The first part of the regular expression was taken directly from gometalinter (see https://github.com/alecthomas/gometalinter/blob/master/linters.go#L236), and I then appended ,\".*\" to additionally grab the line of code of the offending line. Lastly, I excluded ".*Errors unhandled.*fmt.Fprintf.*" to filter out only fmt.Fprintf errors around omitted errors.
Also as a result of this change, gas lint output will now include the offending code.
Closes#590
The main goal of this change is to remove async operations from node manager.
Additionally all of the signals from node manager are moved to status backend.
All of the async operation now will have the following behaviour:
- If node in the correct state exit immediatly without error
- If node not in the correct state exit immediatly with error
- In all other cases spawn a goroutine with wanted operation
- All the progress regarding that operation will be reported
by using signals
- Signals should be handled in once place, which is StatusBackend
There are 2 potentially breaking changes:
- Empty event field will be ommited when Envelope is sent to a client
- All errors will be delivered to a client as an Envelope, previously
some errors (NodeExists, NoRunningNode) were delivered synchronously
Signed-off-by: Dmitry Shulyak <yashulyak@gmail.com>
* fix sync-and-exit logic
* fix reading from doneSync channel
* clean up
* make statusd before using it
* move syncAndStopNode to different file
* change log level in travis chain sync command
* do not use fmt but log
* add progress log and set timeout to 20 mins
* set datadir to .ethereumtest
* fix datadir
Fix failing tests for users w/o access to the ACCOUNT_PASSWORD env variable. Disable e2e public network tests in Travis pull requests. Exclude lib dir from unit tests.
This PR refactors CLI API, removes obsolete commands and splits status code into smaller pieces:
* get rid of subcommands API (no ./status <command>)
* get rid of custom cli app package
* use stdlib flag package for handling command line flags
* move cross-compilation / mobile related code to lib/ package
* move wnode command into separate binary (cmd/node-status, name is subject to discuss)
* remove faucet command as obsolete
* update/add docs/READMES/wikis for new command line flags
It makes statusd code much simpler and smaller, separates concerns (lib, wnode and statusd are different things).
Refactor and little bit clean up Notify api:
Created interface and package "notification" and extracted related code into it
Set dependencies into constructor notificatotion.Manager
Mocks for notificatotion.Manager and FCMClient
Bacis unit tests for Notify and Send
Notify func is now deprecated
Notify users uses new API: message, Payload and a list of tokens
Solves #387 linters was integrated into ci
Added inergration linters to CI
Important changes:
- new linter warnings was fixed
- extracted linters targets into separated mk file
During CI tests non-deterministic failures with "no suitable peers available" happened. Reason is a not finished synchronisation after starting of node.
Added and integrated an EnsureSychronization() almost solved it, but overlapped with new added EnsureNodeSync() in merged develop. Failure stayed, so exchange new algorithm in EsureNodeSync() with former one.
This change moves our e2e tests into a separate package to make room for proper unit and integration tests.
This is Phase 1 described in #371.
Changes:
Makefile has separate directives to run unit/integration tests and e2e tests,
CI runs unit/integration tests first and then e2e tests,
E2e tests are in reliability order, i.e. the least reliable tests are run in the end to be sure that nothing else is broken,
Some tests are fixed or quarantined.
This PR provides a way for status-react to trigger push notifications to contacts whose FCMToken they possess. It thus solves the basic user story as outlined in #326
Now all transactions in both cases are waiting for CompleteTransaction or DiscardTransaction to proceed independently from their destination: upstream of local
* static: updates Whisper test (to work with Geth 1.6.1)
* jail: VM persistence implemented
* jail: sendMessage/showSuggestions minor fixes (to be squashed)
* node: CHT and boot nodes auto-load implemented
* Replaced CHT data file from farazdagi's to tiabc's
* Rewrote config_test.go using testify having reduced it twice in size
* Increased SyncTime and panic timeout in tests
* Fixed test - remove go default test to testify/suite (#207)
* Add flag setup for RPCEnabled and add comment (#225)
* jail: register method handlers before running initial js in jail (#226)
* Console Jail Mod #179 (#228)
* Added ./statusd-data into .gitignore
* Increased log level for the test node from INFO to ERROR
* Add call to loop.Run to evaluate all setTimeout/setIntervals methods. (#208)
* Rebase onto geth1.6.7 (#232)
* Got back sync duration from 60s to 30s, updated bindata.go
- API is async
- Node manager, backend and API modules have more that 90% coverage
- For each level (node manager, backend, public API) random testing
was used to ensure that we don't have race conditions
- node: signals and node reset, fixes#152
- tests update (testify is used)
- node manager refactored, race conditions fixed
- node wrapper has been removed, we rely on go-ethereum node now
- tx queue refactored
- jail refactored
- backend and API introduced
Dmitry Shulyak
Adam Babik
Pedro Pombeiro
Ivan Daniluk
Andrea Maria Piana
Adam Babik
Sebastian Delgado
Adrià Cidre
Andrea Franz
Frank Mueller
Igor Mandrigin
Dmitry Shulyak
Alex Kohler
Caner Çıdam
Denis Makogon
phcurtis
F. Eugene Aumson
Evgeny Danilenko
perissology
Ivan Tomilov
b00ris
Ewetumo Alexander
Oskar Thorén
Marin
Victor Farazdagi