Why make this change?
Currently we only rate limit by number of messages.
This works well if we want to limit a large amount of small messages but
breaks down if sending a smaller amount of large messages.
This commit extends the current code to limit by size as well, setting
the default to 1MB per second, which should be enough.
What has changed?
- Rate limiter for waku only
The index for message was fairly inefficient as it was only using the
cursor, as it was referring to the old `chat_id` field.
This meant that newer messages would be fetched much faster then older
messages.
The index has been changed so that now it includes `local_chat_id`
(which is currently used for filtering), and not using `hide`.
The reason being is that `hide` is a low cardinality index, so there's
no performance benefit to have it in, also it's mostly ignored by the
query planner.
This commit also adds the missing migrations, we generated the file, but
the source was missing, probably I forgot to add them in a rebase. They
have been generated from the migration file, using `RestoreAsset`.
'm trying to add more documentation to parts of the repo that I go into.
I feel this basic history will make it easier to understand why the protocol package is as big as it is compared to the other packages in the repo, and help in understanding its existence.
Before we had two directories `lib/` and `mobile/` that generated
respectively the bindings for desktop and ios/android.
This needed to be kept in sync and there was a fair amount of code
duplication, plus some missing methods on one or the other side.
I have made a change so the whole `lib/` namespace is generated by
parsing the `AST` of `mobile`, and bindings are generated before
compiling.
Incentivisation was an experiment in running an incentivised fleet that
rewarded nodes based on their well behavior. It was heavily influenced
by https://docs.loki.network/ . It is currently not used anymore, so
removing.
We used *Raw method in message_processor as before we had non-Raw method
during the transition from status-react to status-go. This naming is not
meaningful anymore, so I have changed it.
Images are too large to be sent over waku/whisper with the
current PoW (0.002).
This commit lowers the PoW for sending messages to a lower value,
depending on the size.
This means that older clients will not be able to receive messages that
are greater than 50KB (none of the messages we currently send is anywhere close to
that, most of them are less than 1KB).
I would have preferred to set the PoW explicitly for images to be lower,
but it's not trivial as we use `datasync` to send messages and its
interface takes a payload that is to be dispatched and a function to
dispatch, at initialization stage.
This make it difficult (impossible?) to set a different PoW for a
particular message, without changing function signature in datasync
(which is agnostic to the transport used).
So a less cumbersome approach is to just fingerprint on size.
This commit adds support for images in protobuf messages.
The client can specify a path which will be used to load the image
and set the corresponding fields.
This makes the assumption that the RCP server runs on the same machine
as the client and they have access to the same files. This holds
currently for both status-react and status-console-client, we could
revisit and adds an upload if that changes in the future.
This commit lowers the default minimum pow for both Waku & Whisper.
We need to override the value as mobile clients have this already stored
in the database.
This is done in order to support sending of images.
Why make the changes?
Mainly performance, those fields are almost always present in the
database but they are re-calculated on load by the client as it does not
have necessarily access to it.
What has changed?
- Remove `_legacy` persistence namespaces as it's a vestige of the
initial move frmo status-react to status-go
- Pulling chats is now a join with contacts to add contact & alias
see for details : https://dependabot.com/docs/config-file/#allowed_updates
## Why make the change?
@dependabot makes PRs for upgrading dependencies, but it doesn't automatically call `make vendor`, which is a problem because that means any merge for a @dependabot PR will require manual intervention.
I've looked through the @dependabot PR history and there has never been a successfully merged PR. [See here](https://github.com/status-im/status-go/pulls?q=is%3Apr+author%3Aapp%2Fdependabot-preview+is%3Aclosed)
If we are going to use this service we should only be warned of security updates, until we can implement a `make-vendor-bot` to automatically update the vendored files.
## What has changed?
Changed @dependabot settings to only make a PR for security updates.
* fix: close resultsets so we don't leak them
* Refactor browsers/database
To implement PR suggestions and improve code quality.
* Refactor services/permissions/database
To implement PR suggestions and improve code quality.
Co-authored-by: Samuel Hawksby-Robinson <samuel@samyoul.com>
* Added waku README.md, starting text and structure
Resolves GH-1944
* Added api, config and doc details
* Created Docs for waku.const, envelope and events
* Added notes on Filter
* Rewrote the filter section
* Restructured docs to match new versioning structure
* Moved package files out of the main ToC
* Added waku/common/README.md
* Added details to mailserver.go and notes to api.go
* Update to bloomfilter and const common
* Added common/ helpers.go, errors.go and message.go details
* Added common/metrics.go and protocol details
* Added rate_limiter and topic.go details
* fix rate limiter table
* Added Waku Versioning details
* Added Versioned sub-package file details
* Added better detail for Waku Filters
Addressing https://github.com/status-im/status-go/pull/1949#discussion_r419232235
* Added notes about light nodes not being actually new
* Added more detail for Waku node configs and defaults
* Added details to the version message.go section
* Added details to waku/common/rate_limiter
* Minor typo fix in docs
During the move to waku/1 we removed handing of a deprecated format.
aa7f591587 (diff-ea5e44cf3db4a12b3a2a246c7fa39602R290)
Turns out that no client is using the new format and the only live
client is using the deprecated format.
This commit re-introduces the functionality.
Moved sendBundle() from v*/message_response.go to v*/peer.go
Moved TestSendBundle() from v*/message_response_test.go to v*/peer_test.go
Renamed message.go to v*/message_response.go
Renamed message_test.go to v*/message_response_test.go
* Refactor tidy of waku package
* Added deprecation warning on whisper README.md
* Appeasing the lint gods and testing is good
* Place Whisper deprecation warning in the correct package README
:facepalm
* Implementing changes after team feedback
* More offerings to the lint gods
* Remove apparently redundant context params
* Correctly handle concurrent HandlePeer err
* Revert "Remove apparently redundant context params"
This reverts commit 557dbd0d64.
* Added note to waku/api.go about context
* renamed statusoptions and removed unused global
* Removed OnNewP2PEnvelopes() from WakuHost interface
* Matched v1 Peer with new interface sig
Also changed common/helper.go to common/helpers.go
* Formatting of waku tests and some additional error handling
* Changed version to 0.53.0
* Removed redundant type declaration
* Moved TopicToBloom function into a Topic{} method
* Moved GenerateSecureRandomData() into helpers.go
Why make the change?
This implements waku/1 which is a breaking change as waku/0 diverged
from the specs.
What has changed?
- Added v1 namespace
- Changed the testing code to test from the outside rather than stubbing
p2p specific messages, so that any version specific code is under `vx`
namespaces
- Split waku vs waku/x test code
- Added a test suite that can be used for different versions
Things still to do
I kept the tests we have for the versioned aspect of waku, probably we
want to add some or change other to provide the optimal test coverage.
This commit creates a waku/1 namespace and adds the code to it.
No changes in the protocol have been made (i.e waku/1 is the same as
waku/1 in this commit).
* Refactor tidy of waku package
* Added deprecation warning on whisper README.md
* Appeasing the lint gods and testing is good
* Place Whisper deprecation warning in the correct package README
:facepalm
* Implementing changes after team feedback
* More offerings to the lint gods
* Remove apparently redundant context params
* Correctly handle concurrent HandlePeer err
* Revert "Remove apparently redundant context params"
This reverts commit 557dbd0d64.
* Added note to waku/api.go about context
Why make the change?
As discussed previously, the way we will move across versions is to maintain completely separate
codebases and eventually remove those that are not supported anymore.
This has the drawback of some code duplication, but the advantage is that is more
explicit what each version requires, and changes in one version will not
impact the other, so we won't pile up backward compatible code.
This is the same strategy used by `whisper` in go ethereum and is influenced by
https://www.youtube.com/watch?v=oyLBGkS5ICk .
All the code that is used for the networking protocol is now under `v0/`.
Some of the common parts might still be refactored out.
The main namespace `waku` deals with `host`->`waku` interactions (through RPC),
while `v0` deals with `waku`->`remote-waku` interactions.
In order to support `v1`, the namespace `v0` will be copied over, and changed to
support `v1`. Once `v0` will be not used anymore, the whole namespace will be removed.
This PR does not actually implement `v1`, I'd rather get things looked over to
make sure the structure is what we would like before implementing the changes.
What has changed?
- Moved all code for the common parts under `waku/common/` namespace
- Moved code used for bloomfilters in `waku/common/bloomfilter.go`
- Removed all version specific code from `waku/common/const` (`ProtocolVersion`, status-codes etc)
- Added interfaces for `WakuHost` and `Peer` under `waku/common/protocol.go`
Things still to do
Some tests in `waku/` are still testing by stubbing components of a particular version (`v0`).
I started moving those tests to instead of stubbing using the actual component, which increases
the testing surface. Some other tests that can't be easily ported should be likely moved under
`v0` instead. Ideally no version specif code should be exported from a version namespace (for
example the various codes, as those might change across versions). But this will be a work-in-progress.
Some code that will be common in `v0`/`v1` could still be extract to avoid duplication, and duplicated only
when implementations diverge across versions.
* Refactor of waku.doc.go
Moved out all code from docs.go, placed much of the seemingly misc code into much more closely related files, created a dedicated const file for housing all the package consts.
* Moved doc_test to message_test
* Refactored type collision and check on nil err
* message_test.go fmt
* Implemented whisper/doc.go refactor
* import order change to attempt to please the linter.
* Bug fix on whisper_test.go
I also spotted a load of redundant type casting and fixed some of them
If the user deletes/leaves a group chat, the chat is set as not active.
This means that if we are re-invited to the chat it won't be shown to
the user.
This commit changes this behavior so that if we are re-invited to the
chat it is set as active again.
When receiving a message with save a contact in the database in order to
avoid re-calculating image/profile.
This contact is then passed to the client, which can negatively impact
performance.
This commit changes the behavior so that only those contacts that have
some custom fields (have been explicitly added by the user, have been
blocked by the user, have sent a contact request or have a verified ens
name) are passed to the client.
Currently replies to messages are handled in status-react.
This causes some issues with the fact that sometimes replies might come
out of order, they might be offloaded to the database etc.
This commit changes the behavior so that status-go always returns the
replies, and in case a reply comes out of order (first the reply, later
the message being replied to), it will include in the messages the
updated message.
It also adds some fields (RTL,Replace,LineCount) to the database which
were not previously saved, resulting in some potential bugs.
The method that we use to pull replies is currently a bit naive, we just
pull all the message again from the database, but has the advantage of
being simple. It will go through performance testing to make sure
performnace are acceptable, if so I think it's reasonable to avoid some
complexity.
Andrea Maria Piana
Samuel Hawksby-Robinson
Jakub Sokołowski
André Medeiros