This commit enables mailserver cycle logic by default and make a few
changes:
1) Nodes are graylisted instead of being blacklisted for a set amount of
time. The reason is that if we blacklist, any cut in connectivity
might result in long delays before reconnecting, especially on spotty
connections.
2) Fixes an issue on the devp2p server, whereby the node would not
connect to one of the static nodes since all the connection slots
where filled. The fix is a bit inelegant, it always connects to
static nodes, ignoring maxpeers, but it's tricky to get it to work
since the code is clearly not written to select a specific node.
3) Adds support to pinned mailservers
4) Add retries to mailservers requests. It uses a closure for now, I
think we should eventually have a channel etc, but I'd leave that for
later.
By integrating `zxcvbn` module, it has been added a new endpoint to get password strength quality information like Entropy, CrackTime, CrackTimeDisplay, Score, MatchSequence and CalcTime.
Added related dependences.
Closes#4980
Using latest version every time is just asking for weird random issues.
This commit in `gomobile` has made Go `1.16` required:
80cb72e7af
Since we upgarded Go to 1.16 on CI hosts we can now use the recommended
method of installing tools, which is using `go install`.
In addition to that I've pinned other tools that we install to specific
versions, and dropped use of `modvendor` fork thanks to merging of:
https://github.com/goware/modvendor/pull/13
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* feat: obtain external address for rendezvous
If the ext ip returned by geth is 127.0.0.1, it will attempt to obtain the external IP address via rendezvous and use that to register the ens record later
* fix: failing test
* fix: code review, and adding external ip address to logs
* Adding wakunode module
* Adding wakuv2 fleet files
* Add waku fleets to update-fleet-config script
* Adding config items for waku v2
* Conditionally start waku v2 node depending on config
* Adapting common code to use go-waku
* Setting log level to info
* update dependencies
* update fleet config to use WakuNodes instead of BootNodes
* send and receive messages
* use hash returned when publishing a message
* add waku store protocol
* trigger signal after receiving store messages
* exclude linting rule SA1019 to check deprecated packages
* Migrations in place, how to run them?
* Remove down migrations and touch database.go
* Database and Database Test package in place, added functions to get and store app metrics
* make generate output
* Minor bug fix on app metrics insert and select
* Add a validation layer to restrict what can be saved in the database
* Make validation more terse, throw error if schema doesn't exist, expose appmetrics service
* service updates
* Compute all errors before sending them out
* Trying to bring a closjure to appmetrics go
* Expose appmetrics via an api, skip fancy
* Address value as Jason Dawt Rawmasage to ease parsing
* Introduce a buffered chan with magic cap of 8 to minimize writes to DB. Tests for service and API. Also expose GetAppMetrics function.
* Lint issues
* Remove autoincrement, undo waku.json changes, fix error being shadowed, return nil where nil ought to be returned, get rid of buffered channel
* Bump migration number
* Fix API factory usage
* Add comment re:json.RawMessage instead of strings
* Get rid of test vars, throw save error inside the loop
* Update version
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
This fix puts an end to a saga that essentially start during the
Status Prague Meetup at the end of October 2018. At the time we were
experiencing massive issues with `Connecting...` spinners in the app in the
venue we rented. We were pulling our hairs out what to do and we could not
find the cause of the issue at the time.
Three months later I deployed the following change:
https://github.com/status-im/infra-eth-cluster/commit/63a13eed
Which used `iptables` to map the `443` port onto our `30504` Status node port
using `PREROUTING` chain and `REDIRECT` jump in order to fix issues people
have been complaining about when using WiFi networks in various venues:
https://github.com/status-im/status-react/issues/6351
Our thinking when trying to resolve the reported issue assumed that some
networks might block outgoing connections on non-standard ports other than
the usual `80`(HTTP)/`443`(HTTPS) which would disrupt Status connectivity.
While this fix could have indeed helped a few edge cases, what it really
did was cause the Status node to stop seeing actual public IPs of the clients.
But __pure accident__ this change caused the code we inherited from
`go-ethereum` implementation of DevP2P protocol to stop throttling new
incoming connections, because the IP as which they appeared was a
`172.16.0.0/12` network address of the Docker bridge.
The `go-ethereum` code used the `!netutil.IsLAN(remoteIP)` check to
avoid throttling connections from local addresses, which included the
local Docker bridge address:
https://github.com/status-im/status-go/blob/82680830/vendor/github.com/ethereum/go-ethereum/p2p/netutil/net.go#L36
The fix intended to target a small number of networks with fortified
firewall configuration accidentally resolved our issues with
`Connecting...` prompts that our application showed us en masse during
our Prauge Meetup. Part of the reason for that is that venues like that
normally give out local IP addresses and use NAT to translate them onto
the only public IP address they possess.
Since out application is supposed to be usable from within networks
behind NAT like airport WiFi networks for example, it makes no sense to
keep the inbound connection throttle time implemented in `go-ethereum`.
I'm leaving `inboundThrottleTime` in because it's used to calculate
value for `dialHistoryExpiration` in:
`vendor/github.com/ethereum/go-ethereum/p2p/dial.go`
I believe reducing that value one we deploy this change should also
increase the speed with which the Status application is able to reconnect
to a node that was temporarily unavailable, instead waiting the 5*30 seconds.
Research issue: https://github.com/status-im/infra-eth-cluster/issues/35
Signed-off-by: Jakub Sokołowski <jakub@status.im>
There seems to be an issue with version 1.3, querying for topics on
postgres returns
and error:
```
panic: pq: invalid byte sequence for encoding "UTF8"
```
Upgrading pq fixes the issue
¯\_(ツ)_/¯
There was a bug on status-react where it would save filters that were
not listened to.
This commit adds a task to clean up those filters as they might result
in long syncing times.
This commit also returns topics/ranges/mailserves from messenger in
order to make the initialization of the app simpler and start moving
logic to status-go.
It also removes whisper from vendor.
This commit fixes a bug on the mvds library where the nextEpoch would be
incorrectly summed to the retry time, resulting in messages not being
retried, or retried much less frequently the longer the app was running.
It also updates the retry timing to backoff exponentially at multiple of
30 seconds.
## What has changed?
I've introduced to the public binding functionality that will compress and decompress public keys of a variety of encoding and key types. This functionality supports all major byte encoding formats and the following EC public key types:
- `secp256k1` pks
- `bls12-381 g1` pks
- `bls12-381 g2` pks
## Why make the change?
We want shorter public (chat) keys and we want to be future proof and encoding agnostic. See the issue here https://github.com/status-im/status-go/issues/1937
---
* Added basic signature for compresspk and uncompresspk
* Added basic encoding information
* make vendor
* formatted imports for the linter
* Reformatted imports hoping linter likes it
* This linter is capricious
* Added check that the secp256k1 key is valid
* Added test for valid key
* Added multiformat/go-varint dep
* Added public key type handling
* Added key decompression with key type handling
* Added handling for '0x' type indentifying
* Added more robust testing
* Less lint for the linting gods
* make vendor for bls12_381
* Added bls12-381 compression tests
* Added decompress key expected results
* Refactor of typed and untyped keys in tests
* Lint god appeasment
* Refactor of sample public keys
* Implemented bls12-381 decompression
* gofmt
* Renamed decode/encode funcs to be more descriptive
* Added binary bindings for key de/compression
* Refactor of func parameters
gomobile is a bit tempermental using raw bytes as a parameter, so I've decided to use string only inputs and outputs
* gofmt
* Added function documentation
* Moved multiformat de/compression into api/multiformat ns
* Moved multiformat de/compression into api/multiformat ns
* Changed compress to serialize on API
Why make the change?
As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.
This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .
All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.
In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.
This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.
What has changed?
- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`
Things still to do
Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.
Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
* Add status-option code
This commits changes the behavior of waku introducing a new status-code,
`2`, that replaces the current single options codes.
* linting
This resolves a dependency conflict we have with MatterBridge
which was using a newer version of the same package.
This resulted in a JSON marshalling bug that would crash the bridge.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
This commit does a few things:
1) Handle membership updates using protobuf and adds the relevant
endpoints.
2) Store in memory a map of chats + contacts for faster lookups, which
are then flushed to disk on each update
3) Validate incoming messages
Sorry for the large pr, but you know, v1 :)
* Use a single Message type `v1/message.go` and `message.go` are the same now, and they embed `protobuf.ChatMessage`
* Use `SendChatMessage` for sending chat messages, this is basically the old `Send` but a bit more flexible so we can send different message types (stickers,commands), and not just text.
* Remove dedup from services/shhext. Because now we process in status-protocol, dedup makes less sense, as those messages are going to be processed anyway, so removing for now, we can re-evaluate if bringing it to status-go or not.
* Change the various retrieveX method to a single one:
`RetrieveAll` will be processing those messages that it can process (Currently only `Message`), and return the rest in `RawMessages` (still transit). The format for the response is:
`Chats`: -> The chats updated by receiving the message
`Messages`: -> The messages retrieved (already matched to a chat)
`Contacts`: -> The contacts updated by the messages
`RawMessages` -> Anything else that can't be parsed, eventually as we move everything to status-protocol-go this will go away.
This commits add a field (parsedMessage) to the json payload sent to
status-react.
This field is the parsed version of the transit message.
The code is all in dedup, I will re-organize it once we made all the
necesseary changes.
* Add Metadata to messages, expose new messenger methods
This commits modifies deduplicator so that it takes a `StatusMessage`
instead of `WhisperMessage` and also returns a `Metadata` field which is
then passed back by the client when confirming messages, which fixes the
issue we had with not confirming pfs messages.
This commit moves envelopes tracking to status-go.
Post endpoint is not going to track envelopes anymore, as that's taken
care on status-protocol-go side, so this is a breaking change, and
version is updated accordingly.
Adds support for datasync, V1Messages and disabling the discovery topic.
This is a backward compatible change as long as they are not toggled on
(they are not by default).
* add multiaccount support
add multi account ImportPrivateKey and StoreAccount
test derivation from normal keys
* add multiaccount to mobile pkg
* use multiaccount params structs from the mobile pkg
* move multiaccount tests together with the other lib tests
* fix codeclimate warning and temporarily increase methods threshold
* split library_test_utils.go to avoid linter warnings
Richard Ramos
frank
Samuel Hawksby-Robinson
Andrea Maria Piana
Pascal Precht
Noelia
Anthony Laibe
Vitaliy Vlasov
Jakub Sokołowski
Michele Balistreri
Volodymyr Kozieiev
Shivek Khurana
Jonathan Rainville
Roman Volosovskyi
Adam Babik
Pedro Pombeiro
yenda
Adam Babik
Pedro Pombeiro
acolytec3
Andrea Franz