These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history
When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.
If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.
If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.
We will have two components that will rely on this logic:
1. requesting history
If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.
2. confirmation from selected mail server
Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.
I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server
Currently this feature is hidden, as certain changes will be necessary in status-react.
partially implements: https://github.com/status-im/status-go/issues/1285
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.
The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
- Replace command line flags with `-c` config flag. Part of #1180
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
* remove sleeps in WhisperMailboxSuite
* use envelopeFeed to wait for envelopes to be available in group chat test
* remove unsed envelopeTracer in whisper e2e tests
* unsubscribe from envelopeFeed
* use s.addPeerSync instead of WaitForPeerAsync
* mailserver sends envelopes in descending order
* add limit value in mailserver request payload
* mailserver sends messages up to the limit specified in the request
* update Archive method to return key and error
* processRequest returns the next page cursor
* add cursor to mailserver request
* add limit and cursor to request payload
* fix request limit encoding
* wait for request completed event in TrackerSuite/TestRequestCompleted
* add cursor to mailserver response
* fix cursor position in payload
* add e2e test for mail server pagination
* validate mail server response size
* remove old limitReached var
* fix lint warnings
* add whisper patch
* fix tests after rebase
* check all return values to avoid lint warnings
* check that all messages have been retrieved after 2 paginated requests
* fix lint warnings
* rename geth patch
* merge mailserver patches into one
* add last envelope hash to mailserver response and EventEnvelopeAvailable event
* update whisper patch
* add docs to MailServerResponse
* update whisper patch
* fix tests and lint warnings
* send mailserver response data on EventMailServerRequestCompleted signal
* update tracker tests
* optimise pagination test waiting for mailserver to archive only before requesting
* rollback mailserver interface changes
* refactoring and docs changes
* fix payload size check to determine if a limit is specified
* add more docs to the processRequest method
* add constants for request payload field lengths
* add const noLimits to specify that limit=0 means no limits
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper
* send p2pRequestResponseCode from mailserver
* send p2p message response to after sending all historic messages
* mailserver sends `whisper.NewSentMessage` as response
* add mailserver Client and p2pRequestAckCode watchers
* send event with envelopeFeed when p2pRequestAckCode is received
* test request completed event in tracker
* rename mailserver response events and code to RequestCompleteCode
* wait for mailserver response in e2e test
* use SendHistoricMessageResponse method name for mailserver response
* fix lint warnings
* add mailserver request expiration
* send mailserver response without envelope
* add `ttl` to Request struct in shhext_requestMessages
* test that tracker calls handler.MailServerRequestExpired
* add geth patch
* rename TTL to Timeout
* split tracker.handleEvent in multiple methods
This change adds adds an ability to use different source of time for whisper:
when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.
So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.
In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.
Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.
Closes: #687
Every peer must be subscribed to the topic that is used to send messages.
In the test Alice was communicating with Bob and Charlie over custom topic, but
that topic wasn't added to a bloom filter, thus a certain flake was possible.
Normally it wasn't causing problems because syncAllowance in whisper, which is 10s:
- we set bloom filter to all zeros
- but we still will accept all envelopes for 10s
- in case we send first envelope into such channel after sync allowance - we will get an error such
that envelope doesn't match the bloom filter
* Add RequestMessage to sshext
* E2E tests now use shhext_requestMessages
* Typo in comment
* Enhanced maintainability
* Drop former mailservice
* Code reorg after review
* Fix missed changes after update to 1.8.5
* Rebase on 1.8.5
* Remove outdated patches and apply all others
* Use shh_post that returns hash
* Use bloom filter for request to mailserver
* Remove tests for sending messages without subbing first
* Fix deadlock in ethdb
* Expect null if receipt is not yet created
* Subscribe to messages before sending them in whisper test
* Start enabling to test Mainnet
* Minor corrections found during code scan
* Set mainnet blocker in E2E again
* Introduced securing of mainnet transaction tests
* Fix typing error
* Typo led to follow-up errors
* Linter problem
* Change to individual test skips after review
* More flexible skip control for mainnet and status chain
* Fix double space
* Change of skipping method and fixing wrong skipped networks
This change will greatly simplify writing unit tests when a node is required but data persistence is irrelevant.
I also Introduced some refactoring and unit tests for `StatusNode`.
* Implement shh api extension that allows to confirm that message is sent
* Add a patch
* Fix linter
* Add readme
* Add tests for tracker
* Address review
- [x] [#797] : Remove unused methods PopulateStaticPeers, ReconnectStaticPeers, removeStaticPeers, removePeer
- [x] [#797] : Rename node.Manager to node. StatusNode and simplify its public api
- [x] [#797] : Rename all references to nodeManager to statusNode
I am not 100% percent sure what is happening but it seems that
newMessageFilter is async operation, result of this operation is assumed
to be used in callbacks. All other tests are doing at least 1 io operation
in between creating a filter and posting a message, and it must be enough
for newMessageFilter to complete.
Setting higher GOMAXPROCS allows Otto vm to execute received io requests
immediatly. While lower number of processes may result in events re-ordering.
https://github.com/ethereum/web3.js/blob/develop/lib/web3/methods/shh.js#L39-L41
I tested this change with GOMAXPROCS=1 and it passes consistently.
Adam Babik
Dmitry Shulyak
Pedro Pombeiro
Adam Babik
Ivan Daniluk
Andrea Franz
Adrià Cidre
Sebastian Delgado
Dmitry Shulyak
Frank Mueller
Igor Mandrigin
Evgeny Danienko