Move settings table schema from a key-value store to a one row table with many columns.
We now save the first row with initial data in saveAccountAndLogin and follow up saveSetting calls are only saving one setting at a time.
Co-authored-by: Adam Babik <a.babik@designfortress.com>
Account's address was used as a primary key in accounts db and as a
deterministic id of an account in some API calls. Also it was used as a
part of the name of the account specific database. This revealed some
extra information about the account and wasn't necessary.
At first the hash of the address was planned to be used as a
deterministic id, but we already have a keyUid which is calculated as
sha256 hash of account's public key and has similar properties:
- it is deterministic
- doesn't reveal accounts public key or address in plain
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`
* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.
* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.
* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
* WIP accounts implementation
* Accounts datasore and changes to status mobile API
* Add library changes and method to update config
* Handle error after account selection
* Add two methods to start account to backend
* Use encrypted database for settings and add a service for them
* Resolve linter warning
* Bring back StartNode StopNode for tests
* Add sub accounts and get/save api
* Changes to accounts structure
* Login use root address and fetch necessary info from database
* Cover accounts store with tests
* Refactor in progress
* Initialize status keystore instance before starting ethereum node
* Rework library tests
* Resolve failures in private api test and send transaction test
* Pass pointer to initialized config to unmarshal
* Use multiaccounts/accounts naming consistently
Multiaccount is used as a login identifier
Account references an address and a key, if account is not watch-only.
* Add login timestamp stored in the database to accounts.Account object
* Add photo-path field for multiaccount struct
* Add multiaccoutns rpc with updateAccount method
Update to any other account that wasn't used for login will return an error
* Fix linter in services/accounts
* Select account before starting a node
* Save list of accounts on first login
* Pass account manager to accounts service to avoid selecting account before starting a node
* Add logs to login with save and regualr login
This commit moves envelopes tracking to status-go.
Post endpoint is not going to track envelopes anymore, as that's taken
care on status-protocol-go side, so this is a breaking change, and
version is updated accordingly.
* multi-account login and signing
put methods count threshold back to 20
* validate login params
* refactoring
* use common.Address
* remove unused var in test
* add multiaccount support
add multi account ImportPrivateKey and StoreAccount
test derivation from normal keys
* add multiaccount to mobile pkg
* use multiaccount params structs from the mobile pkg
* move multiaccount tests together with the other lib tests
* fix codeclimate warning and temporarily increase methods threshold
* split library_test_utils.go to avoid linter warnings
* Store tx and receipt in db and cast it to TransferView on read
* Store Log instead of log index
* Use contract from log and bring back address field
* Add tx status and id fields
[services/wallet] Several changes in API after feedback
- Timestamp from block header stored in blocks table and added to each transfers
- From field is computed from signature to offload this computation from client side
- `history` event is added back, so that client can ignore historical blocks when watching
only for new blocks
- block number and timestamp are marshalled in hex. consistent with ethereum data structures
This commit add topic negotiation to the protocol.
On receiving a message from a client with version >= 1, we will generate
a shared key using Diffie-Hellman. We will record also which
installationID has sent us a message.
This key will be passed back to the above layer, which will then use to
start listening to a whisper topic (the `chat` namespace has no
knowledge of whisper).
When sending a message to a set of InstallationIDs, we check whether we
have agreed on a topic with all of them, and if so, we will send on this
separate topic, otherwise we fallback on discovery.
This change is backward compatible, as long as there is no downgrade of
the app on the other side.
A few changes:
* Factored out the DB in a separate namespace as now it is
being used by multiple services (TopicService and EncryptionService).
* Factored out multidevice management in a separate namespace
* Moved all the test to test the whole protoocl rather than just the encryption service
* Moved all the filter management in status-go
* Split shhext.tracker into envelopes and mail monitors
* Send envelopes on every new attempt to deliver a message
* Re-send user payloads if previous envelopes weren't acknowledged
* Remove debug api across the codebase
* select account decrypting wallet and chat keys
* adapt account tests to use chat and wallet account/keys
* fix tests using chat address
* changes after review
* fix status service e2e tests
* add account.Info struct returned when creating and recovering an account
* use s.EqualValues to compare recovered accounts
* return Info instead of *Info
* add both address and walletAddress to responses
* Update lib/types.go
Co-Authored-By: gravityblast <andrea@gravityblast.com>
* Update lib/types.go
Co-Authored-By: gravityblast <andrea@gravityblast.com>
* update comment to fix lint
* use keyStore.ImportExtendedKeyForPurpose when creating keys
* rename selectecAccount to selectedWalletAccount
* add selectedChatAccount
* update e2e test to check that injected whisper key is the chat public key
* update TestSelectedAccountOnRestart to check that chat key is used in whisper
* use chat account in api/backend
* update mocks
* temporarily update VERSION to build a release
* check that chat/wallet keys are the same in a different test
* add account test to check that wallet and chat keys are the same
* test only that the chat key is injected in whisper
* put back the right VERSION
These changes add a support for syncing data between two Mail Servers. It does not give an easy way to start syncing. This will be solved in the next PR.
This change implements connection manager that monitors 3 types of events:
1. update of the selected mail servers
2. disconnect from a mail server
3. errors for requesting mail history
When selected mail servers provided we will try to connect with as many as possible, and later disconnect the surplus. For example if we want to connect with one mail server and 3 were selected, we try to connect with all (3), and later disconnect with 2. It will to establish connection with live mail server faster.
If mail server disconnects we will choose any other mail server from the list of selected. Unless we have only one mail server. In such case we don't have any other choice and we will leave things as is.
If request for history was expired we will disconnect such peer and try to find another one. We will follow same rules as described above.
We will have two components that will rely on this logic:
1. requesting history
If target peer is provided we will use that peer, otherwise we will request history from any selected mail server that is connected at the time of request.
2. confirmation from selected mail server
Confirmation from any selected mail server will bee used to send a feedback that envelope was sent.
I will add several extensions, but probably in separate PRs:
1. prioritize connection with mail server that was used before reboot
2. disconnect from mail servers if history request wasn't expired but failed.
3. wait some time in RequestsMessage RPC to establish connection with any mail server
Currently this feature is hidden, as certain changes will be necessary in status-react.
partially implements: https://github.com/status-im/status-go/issues/1285
This commit updates geth to 1.8.17 and adds a possibility to enable metrics during compilation time.
The cascade of issues forced us to upgrade geth to 1.8.17 in order to allow enabling metrics during compilation time. 1.8.17 introduced `NodeID` refactoring and `enode` package which affected our peers pool and integration with Discovery V5.
* Implement subscriptions and filtering
* Add e2e test with log filter polling logs from EVM with clique backend
* Apply review comments
* Move devnode to t/devtests to avoid cycle in imports
- Replace command line flags with `-c` config flag. Part of #1180
- Convert node config private keys to hex-encoded string versions.
- Remove `GenerateConfig` from library.
- Remove unused `FirebaseConfig` from library.
- Fix loading of `config/status-chain-genesis.json` in non-dev machines.
In the current codebase if you want to use different keys (or any
other configuration) for tests it is required to place them at a
certain path in static folder, bundle them with go-bindata and only
then run tests. This is simply unnecessary.
This commit instead directs loads configurations from file without
relying on go-bindata
Remove `PendingSignRequests` queue from the sign module.
This closes#1027 by removing the pending sign requests queue dependency from the SendTransaction, SignMessage and Recover.
* remove sleeps in WhisperMailboxSuite
* use envelopeFeed to wait for envelopes to be available in group chat test
* remove unsed envelopeTracer in whisper e2e tests
* unsubscribe from envelopeFeed
* use s.addPeerSync instead of WaitForPeerAsync
* mailserver sends envelopes in descending order
* add limit value in mailserver request payload
* mailserver sends messages up to the limit specified in the request
* update Archive method to return key and error
* processRequest returns the next page cursor
* add cursor to mailserver request
* add limit and cursor to request payload
* fix request limit encoding
* wait for request completed event in TrackerSuite/TestRequestCompleted
* add cursor to mailserver response
* fix cursor position in payload
* add e2e test for mail server pagination
* validate mail server response size
* remove old limitReached var
* fix lint warnings
* add whisper patch
* fix tests after rebase
* check all return values to avoid lint warnings
* check that all messages have been retrieved after 2 paginated requests
* fix lint warnings
* rename geth patch
* merge mailserver patches into one
* add last envelope hash to mailserver response and EventEnvelopeAvailable event
* update whisper patch
* add docs to MailServerResponse
* update whisper patch
* fix tests and lint warnings
* send mailserver response data on EventMailServerRequestCompleted signal
* update tracker tests
* optimise pagination test waiting for mailserver to archive only before requesting
* rollback mailserver interface changes
* refactoring and docs changes
* fix payload size check to determine if a limit is specified
* add more docs to the processRequest method
* add constants for request payload field lengths
* add const noLimits to specify that limit=0 means no limits
* refactor TestRequestMessageFromMailboxAsync to use s.requestHistoricMessages helper
* send p2pRequestResponseCode from mailserver
* send p2p message response to after sending all historic messages
* mailserver sends `whisper.NewSentMessage` as response
* add mailserver Client and p2pRequestAckCode watchers
* send event with envelopeFeed when p2pRequestAckCode is received
* test request completed event in tracker
* rename mailserver response events and code to RequestCompleteCode
* wait for mailserver response in e2e test
* use SendHistoricMessageResponse method name for mailserver response
* fix lint warnings
* add mailserver request expiration
* send mailserver response without envelope
* add `ttl` to Request struct in shhext_requestMessages
* test that tracker calls handler.MailServerRequestExpired
* add geth patch
* rename TTL to Timeout
* split tracker.handleEvent in multiple methods
Some operations (like deploying contracts) require filter APIs to work.
Since these operations aren't supported on Infura anymore, and we don't
run LES, a separate implemenation of filters is required.
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
This change adds adds an ability to use different source of time for whisper:
when envelope is created it is used to set expiry
to track when envelope needs to be expired
This time is then used to check validity of the envelope when it is received. Currently If we receive an envelope that is sent from future - peer will get disconnected. If envelope that was received has an expiry less then now it will be simply dropped, if expiry is less than now + 10*2 seconds peer will get dropped.
So, it is clear that whisper depends on time. And any time we get a skew with peers that is > 20s reliability will be grealy reduced.
In this change another source of time for whisper will be used. This time source will use ntp servers from pool.ntp.org to compute offset. When whisper queries time - this offset will be added/substracted from current time.
Query is executed every 2 mins, queries 5 different servers, cut offs min and max and the computes mean value. pool.ntp.org is resolved to different servers and according to documentation you will rarely hit the same.
Closes: #687
Every peer must be subscribed to the topic that is used to send messages.
In the test Alice was communicating with Bob and Charlie over custom topic, but
that topic wasn't added to a bloom filter, thus a certain flake was possible.
Normally it wasn't causing problems because syncAllowance in whisper, which is 10s:
- we set bloom filter to all zeros
- but we still will accept all envelopes for 10s
- in case we send first envelope into such channel after sync allowance - we will get an error such
that envelope doesn't match the bloom filter
* add LogEnabled attribute to NodeConfig, used in the call from status-react
* fix use of OverrideRootLogin in tests
* enable logger in tests based on LogLevel
* move LogEnabled before LogFile
* Add RequestMessage to sshext
* E2E tests now use shhext_requestMessages
* Typo in comment
* Enhanced maintainability
* Drop former mailservice
* Code reorg after review
* Fix missed changes after update to 1.8.5
* Rebase on 1.8.5
* Remove outdated patches and apply all others
* Use shh_post that returns hash
* Use bloom filter for request to mailserver
* Remove tests for sending messages without subbing first
* Fix deadlock in ethdb
* Expect null if receipt is not yet created
* Subscribe to messages before sending them in whisper test
* Add default peer limits configuration
If discovery is enabled for a given cluster - we will set a default
expected number of peers for each enabled service. For example:
- if cluster is rinkeby has a discovery enabled we will
check which services are enabled
- if whisper is enabled we will set min and max limits by default
- if les is enabled and infura is not used we will set limits too
When statusd is used - configuration must be provided using configuration
supported by statusd.
* Fix deadlock in les peer set
* Start enabling to test Mainnet
* Minor corrections found during code scan
* Set mainnet blocker in E2E again
* Introduced securing of mainnet transaction tests
* Fix typing error
* Typo led to follow-up errors
* Linter problem
* Change to individual test skips after review
* More flexible skip control for mainnet and status chain
* Fix double space
* Change of skipping method and fixing wrong skipped networks
This change will greatly simplify writing unit tests when a node is required but data persistence is irrelevant.
I also Introduced some refactoring and unit tests for `StatusNode`.
* Implement shh api extension that allows to confirm that message is sent
* Add a patch
* Fix linter
* Add readme
* Add tests for tracker
* Address review
We need to be able to sign more than just transactions to make DApps
work properly. This change separates signing requests from the
transactions and make it more general to prepare to intoduce different
types of signing requests.
This change is designed to preserve status APIs, so it is
backward-comparible with the current API bindings.
- [x] [#797] : Remove unused methods PopulateStaticPeers, ReconnectStaticPeers, removeStaticPeers, removePeer
- [x] [#797] : Rename node.Manager to node. StatusNode and simplify its public api
- [x] [#797] : Rename all references to nodeManager to statusNode
* Rename bootnode to static peers
Also add some groupings between types.
* Remove not needed genesisHash
* More cleanup of bootnodes with static peers
* Add option of cluster configuration file
* New generated bindata.go
* Changes after npm install
* Add argument for cluster configuration file
* Add test for dynamic cluster loading
Not yet sure with name "cluster config".
* Solved conflicts
* Renaming of static peers
* Remove static peers population
* Missing argument for config
* Renaming of static peers to boot nodes for consistency
* Fix of name change
* Cluster config is now cluster data
* Load static nodes from configuration
* Final renaming of var for file content
* Add debug log to test package t
* Add comments for test restrictions
* Add explicit errors for mainnet
* Panic instead of error in case of mainnet during tests
* Now panic with illegal network
* Validate the compiler flag for mainnet
* Made test for unset mainnet explicit
* Forgot to use my flag
* Linter has troubles with unused field
* Add default network id
In e2e tests there's no default anymore in go code. So ensure in Makefile.
* Fix dumb syntax error
* CI needs build with Mainnet
* Explicit rebuild only for tests
* No more separation of mainnet and testnet builds
* Build flags now set in Makefile
I am not 100% percent sure what is happening but it seems that
newMessageFilter is async operation, result of this operation is assumed
to be used in callbacks. All other tests are doing at least 1 io operation
in between creating a filter and posting a message, and it must be enough
for newMessageFilter to complete.
Setting higher GOMAXPROCS allows Otto vm to execute received io requests
immediatly. While lower number of processes may result in events re-ordering.
https://github.com/ethereum/web3.js/blob/develop/lib/web3/methods/shh.js#L39-L41
I tested this change with GOMAXPROCS=1 and it passes consistently.
* Update project to use Whisper v6. Part of #638
* Revert "Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #665" - this reverts commit 6aefb4c8fd02dbcfffac6b69e8bb22b13ef86b6b.
* Enable light mode on Whisper v6 for non-mail servers. Part of #638
* Fix race condition in whisperv6/peer.go. Part of #665 (PR already accepted upstream for 1.8.2)
* Update bootnode addresses in staticnodes.json. Part of #638
* Add `shh.lightclient` flag and tests for bloom filter setting logic. Part of #638
* Move MakeTestNodeConfig to utils. Part of #638
* Reduce PoW in `whisper_jail_test.go` to fix flaky test. Part of #638
* Update `github.com/ethereum/go-ethereum` package to 1.8.1 branch. Part of #638
* Fix code due to some signature changes. Part of #638
* use upstream for whisper backend
* Add patch to downgrade usage of Whisper v6 to v5 in some geth 1.8.1 vendor files. Part of #638
* Take into account the DNS rebinding protection introduced in 1.8.0 by adding exception for localhost. Part of #638
* Add patches required for cross-compiled builds starting with geth 1.8.0. Only applied during build. Part of #638
* Update expected JSON result in `TestRegressionGetTransactionReceipt()` and `TestCallRawResultGetTransactionReceipt()`. Part of #665
* Fix some failing e2e tests. Part of #638
* Address comments in PR #702. Part of #638
Test waits till synchronization is started, then interrupts network connection, wait for a failure event and restore network connection, confirming that after the connection is restored synchronization will proceed.
Network disconnect is introduced by removing default gateway, easily reversible condition.
On my local machine it takes 30 seconds for peers to reconnect after connectivity is restored. As you guess this is not an accident, and there is 30 seconds timeout for dial expiration. This dial expiration is used in p2p.Server to guarantee that peers are not dialed too often.
Additionally I added small script to Makefile to run such tests in docker environment, usage example:
```
make docker-test ARGS="./t/destructive/ -v -network=4"
```
Improve patcher tool:
* fix broken `-p` flag,
* rename it to `-b`,
* add a new `-p` flag that allows specifying the source of the patches, so that it can be used for more than just go-ethereum patches,
* make it work correctly independently of the order that options are specified in the command line.
Adam Babik
Andrea Maria Piana
Pedro Pombeiro
Adam Babik
yenda
André Medeiros
Roman Volosovskyi
dmitry
Andrea Franz
Ivan Daniluk
Kishan Sagathiya
Sebastian Delgado
Adam Babik
Adrià Cidre
Igor Mandrigin
Dmitry Shulyak
Frank Mueller
Evgeny Danienko
Alex Kohler