* Replace mclock with time in peer pools
we used mclock as golang before 1.9 did not support monotonic clocks,
https://github.com/gavv/monotime, it does now https://golang.org/pkg/time/
so we can fallback on the system implementation, which will return
nanoseconds with a resolution that is system dependent.
* Handle case where peer have same discovered time
In some system the resolution of the clock is not high enough so
multiple peers are added on the same nanosecond.
This result in the peer just added being immediately removed.
This code adds a check making sure we don't assume that a peer is added.
Another approach would be to make sure to include the peer in the list,
so prevent the peer just being added to be evicted, but it's slightly
more complicated and the resolution is generally accurate enough for our
purpose so that peers will be fresh enough if they have the same
discovered time.
It also adds a regression test, I had to use an interface to stub the
clock.
Fixes: https://github.com/status-im/nim-status-client/issues/522
* bump version to 0.55.3
Geth decided to link to this place here https://github.com/ethereum/whisper, and it isn't a very welcoming README. Hopefully this points people in the right direction :)
## What has changed?
I've introduced to the public binding functionality that will compress and decompress public keys of a variety of encoding and key types. This functionality supports all major byte encoding formats and the following EC public key types:
- `secp256k1` pks
- `bls12-381 g1` pks
- `bls12-381 g2` pks
## Why make the change?
We want shorter public (chat) keys and we want to be future proof and encoding agnostic. See the issue here https://github.com/status-im/status-go/issues/1937
---
* Added basic signature for compresspk and uncompresspk
* Added basic encoding information
* make vendor
* formatted imports for the linter
* Reformatted imports hoping linter likes it
* This linter is capricious
* Added check that the secp256k1 key is valid
* Added test for valid key
* Added multiformat/go-varint dep
* Added public key type handling
* Added key decompression with key type handling
* Added handling for '0x' type indentifying
* Added more robust testing
* Less lint for the linting gods
* make vendor for bls12_381
* Added bls12-381 compression tests
* Added decompress key expected results
* Refactor of typed and untyped keys in tests
* Lint god appeasment
* Refactor of sample public keys
* Implemented bls12-381 decompression
* gofmt
* Renamed decode/encode funcs to be more descriptive
* Added binary bindings for key de/compression
* Refactor of func parameters
gomobile is a bit tempermental using raw bytes as a parameter, so I've decided to use string only inputs and outputs
* gofmt
* Added function documentation
* Moved multiformat de/compression into api/multiformat ns
* Moved multiformat de/compression into api/multiformat ns
* Changed compress to serialize on API
Why make this change?
Currently we only rate limit by number of messages.
This works well if we want to limit a large amount of small messages but
breaks down if sending a smaller amount of large messages.
This commit extends the current code to limit by size as well, setting
the default to 1MB per second, which should be enough.
What has changed?
- Rate limiter for waku only
The index for message was fairly inefficient as it was only using the
cursor, as it was referring to the old `chat_id` field.
This meant that newer messages would be fetched much faster then older
messages.
The index has been changed so that now it includes `local_chat_id`
(which is currently used for filtering), and not using `hide`.
The reason being is that `hide` is a low cardinality index, so there's
no performance benefit to have it in, also it's mostly ignored by the
query planner.
This commit also adds the missing migrations, we generated the file, but
the source was missing, probably I forgot to add them in a rebase. They
have been generated from the migration file, using `RestoreAsset`.
'm trying to add more documentation to parts of the repo that I go into.
I feel this basic history will make it easier to understand why the protocol package is as big as it is compared to the other packages in the repo, and help in understanding its existence.
Before we had two directories `lib/` and `mobile/` that generated
respectively the bindings for desktop and ios/android.
This needed to be kept in sync and there was a fair amount of code
duplication, plus some missing methods on one or the other side.
I have made a change so the whole `lib/` namespace is generated by
parsing the `AST` of `mobile`, and bindings are generated before
compiling.
Incentivisation was an experiment in running an incentivised fleet that
rewarded nodes based on their well behavior. It was heavily influenced
by https://docs.loki.network/ . It is currently not used anymore, so
removing.
We used *Raw method in message_processor as before we had non-Raw method
during the transition from status-react to status-go. This naming is not
meaningful anymore, so I have changed it.
Images are too large to be sent over waku/whisper with the
current PoW (0.002).
This commit lowers the PoW for sending messages to a lower value,
depending on the size.
This means that older clients will not be able to receive messages that
are greater than 50KB (none of the messages we currently send is anywhere close to
that, most of them are less than 1KB).
I would have preferred to set the PoW explicitly for images to be lower,
but it's not trivial as we use `datasync` to send messages and its
interface takes a payload that is to be dispatched and a function to
dispatch, at initialization stage.
This make it difficult (impossible?) to set a different PoW for a
particular message, without changing function signature in datasync
(which is agnostic to the transport used).
So a less cumbersome approach is to just fingerprint on size.
This commit adds support for images in protobuf messages.
The client can specify a path which will be used to load the image
and set the corresponding fields.
This makes the assumption that the RCP server runs on the same machine
as the client and they have access to the same files. This holds
currently for both status-react and status-console-client, we could
revisit and adds an upload if that changes in the future.
This commit lowers the default minimum pow for both Waku & Whisper.
We need to override the value as mobile clients have this already stored
in the database.
This is done in order to support sending of images.
Why make the changes?
Mainly performance, those fields are almost always present in the
database but they are re-calculated on load by the client as it does not
have necessarily access to it.
What has changed?
- Remove `_legacy` persistence namespaces as it's a vestige of the
initial move frmo status-react to status-go
- Pulling chats is now a join with contacts to add contact & alias
see for details : https://dependabot.com/docs/config-file/#allowed_updates
## Why make the change?
@dependabot makes PRs for upgrading dependencies, but it doesn't automatically call `make vendor`, which is a problem because that means any merge for a @dependabot PR will require manual intervention.
I've looked through the @dependabot PR history and there has never been a successfully merged PR. [See here](https://github.com/status-im/status-go/pulls?q=is%3Apr+author%3Aapp%2Fdependabot-preview+is%3Aclosed)
If we are going to use this service we should only be warned of security updates, until we can implement a `make-vendor-bot` to automatically update the vendored files.
## What has changed?
Changed @dependabot settings to only make a PR for security updates.
* fix: close resultsets so we don't leak them
* Refactor browsers/database
To implement PR suggestions and improve code quality.
* Refactor services/permissions/database
To implement PR suggestions and improve code quality.
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
* Added waku README.md, starting text and structure
Resolves GH-1944
* Added api, config and doc details
* Created Docs for waku.const, envelope and events
* Added notes on Filter
* Rewrote the filter section
* Restructured docs to match new versioning structure
* Moved package files out of the main ToC
* Added waku/common/README.md
* Added details to mailserver.go and notes to api.go
* Update to bloomfilter and const common
* Added common/ helpers.go, errors.go and message.go details
* Added common/metrics.go and protocol details
* Added rate_limiter and topic.go details
* fix rate limiter table
* Added Waku Versioning details
* Added Versioned sub-package file details
* Added better detail for Waku Filters
Addressing https://github.com/status-im/status-go/pull/1949#discussion_r419232235
* Added notes about light nodes not being actually new
* Added more detail for Waku node configs and defaults
* Added details to the version message.go section
* Added details to waku/common/rate_limiter
* Minor typo fix in docs
During the move to waku/1 we removed handing of a deprecated format.
aa7f591587 (diff-ea5e44cf3db4a12b3a2a246c7fa39602R290)
Turns out that no client is using the new format and the only live
client is using the deprecated format.
This commit re-introduces the functionality.
Moved sendBundle() from v*/message_response.go to v*/peer.go
Moved TestSendBundle() from v*/message_response_test.go to v*/peer_test.go
Renamed message.go to v*/message_response.go
Renamed message_test.go to v*/message_response_test.go
* Refactor tidy of waku package
* Added deprecation warning on whisper README.md
* Appeasing the lint gods and testing is good
* Place Whisper deprecation warning in the correct package README
:facepalm
* Implementing changes after team feedback
* More offerings to the lint gods
* Remove apparently redundant context params
* Correctly handle concurrent HandlePeer err
* Revert "Remove apparently redundant context params"
This reverts commit 557dbd0d64.
* Added note to waku/api.go about context
* renamed statusoptions and removed unused global
* Removed OnNewP2PEnvelopes() from WakuHost interface
* Matched v1 Peer with new interface sig
Also changed common/helper.go to common/helpers.go
* Formatting of waku tests and some additional error handling
* Changed version to 0.53.0
* Removed redundant type declaration
* Moved TopicToBloom function into a Topic{} method
* Moved GenerateSecureRandomData() into helpers.go
Why make the change?
This implements waku/1 which is a breaking change as waku/0 diverged
from the specs.
What has changed?
- Added v1 namespace
- Changed the testing code to test from the outside rather than stubbing
p2p specific messages, so that any version specific code is under `vx`
namespaces
- Split waku vs waku/x test code
- Added a test suite that can be used for different versions
Things still to do
I kept the tests we have for the versioned aspect of waku, probably we
want to add some or change other to provide the optimal test coverage.
This commit creates a waku/1 namespace and adds the code to it.
No changes in the protocol have been made (i.e waku/1 is the same as
waku/1 in this commit).
* Refactor tidy of waku package
* Added deprecation warning on whisper README.md
* Appeasing the lint gods and testing is good
* Place Whisper deprecation warning in the correct package README
:facepalm
* Implementing changes after team feedback
* More offerings to the lint gods
* Remove apparently redundant context params
* Correctly handle concurrent HandlePeer err
* Revert "Remove apparently redundant context params"
This reverts commit 557dbd0d64.
* Added note to waku/api.go about context
Why make the change?
As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.
This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .
All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.
In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.
This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.
What has changed?
- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`
Things still to do
Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.
Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
* Refactor of waku.doc.go
Moved out all code from docs.go, placed much of the seemingly misc code into much more closely related files, created a dedicated const file for housing all the package consts.
* Moved doc_test to message_test
* Refactored type collision and check on nil err
* message_test.go fmt
* Implemented whisper/doc.go refactor
* import order change to attempt to please the linter.
* Bug fix on whisper_test.go
I also spotted a load of redundant type casting and fixed some of them
Andrea Maria Piana
Oskar Thorén
Roman Volosovskyi
Samuel Hawksby-Robinson
Jakub Sokołowski
André Medeiros