This commit completely remove transit for group chats. All the
processing is now done in status-go.
Also introuduces parsing and handling of mentions, needed so that system
messages can be easily built in status-go.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Account's address was used as a primary key in accounts db and as a
deterministic id of an account in some API calls. Also it was used as a
part of the name of the account specific database. This revealed some
extra information about the account and wasn't necessary.
At first the hash of the address was planned to be used as a
deterministic id, but we already have a keyUid which is calculated as
sha256 hash of account's public key and has similar properties:
- it is deterministic
- doesn't reveal accounts public key or address in plain
This commit moves all the processing of messages to status-go.
Messages are going arrive to status-react already saved an processed.
Receiving/sending/retrieving from db is now using the same identical
structure. The only processing left in status-react is to mark the
messages as seen and update the unviewed count locally (only
status-react knows whether the count should be updated).
Partially remove commands as well as won't be used anymore.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- if not mailserver was actively selected by user,
use rpc call to get latency for known mailservers
and use the best one
- this happens when `set-current-mailserver` is called which happens
in `change-mailserver` when user unpins his preferred mailserver and when
there's been too many failed attemps to fetch messages or to connect to
then current mailserverm as well as when user logs in.
Signed-off-by: yenda <eric@status.im>
Fixes: https://github.com/status-im/trailofbits-audit/issues/47
Fixes: https://github.com/status-im/trailofbits-audit/issues/46
Fixes: https://github.com/status-im/trailofbits-audit/issues/44
Fixes: https://github.com/status-im/security-reports/issues/13
Fixes: https://github.com/status-im/security-reports/issues/5
Fixes: https://github.com/status-im/status-react/issues/8995
This commits re-introduce rendering of markdown text and implent a few
changes:
1) Parsing of the message content is now in status-go, this includes
markdown, line-count, and rtl. Parsing is not nested, as there's some
rendering degradation involved as we nest components, unclear exactly if
it's react-native or clojure, haven't looked too deeply into it.
2) Emojii type messages are not parsed on the sending side, not the
receiving one, using the appropriate content-type
3) Fixes a few issues with chat input rendering, currrently we use
`chats/current-chat` subscription which is very heavy and should not be
used unless necessary, and means that
any change to chat will trigger a re-render, which caused re-rendering
of input container on each received message. Also to note that
input-container is fairly heavy to render, and it's rendered twice at
each keypress on input.
The inline markdow supported is:
*italic* or _italic_
**bold** or __bold__
`inline code`
http://test.com links
\#status-tag
The block markdown supported is:
\# Headers
```
code blocks
```
> Quotereply
The styling is very basic at the moment, but can be improved.
Adding other markdown (photo,mentions) is straightforward and should
come at little performance cost (unless the component to render is
heavy, i.e a photo for example).
There are some behavioral changes with this commit:
1) Links are only parsed if starting with http:// or https://, meaning that
blah.com won't be parsed, nor www.test.com. This behavior is consistent
with discord for example and allows faster parsing at little expense to
ser experience imo. Fixes a few security issues as well.
2) Content is not anymore capped (regression), that's due to the fact that
before we only rendered text and react-native allowed us easily to limit
the number of lines, but adding markdown support means that this
strategy is not viable anymore. Performance of rendering don't see to be
very much impacted by this, I would re-introduce it if necessary, but
I'd rather do that in a separate PR.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Fixes#9433 : this was due to the fact that `.-response-to` was
returning nil, because of the dash in the name, instead in this cases
`(aget .. "response-to")` should be used.
Fixes#9431 : This was a left-over from the move from message-groups to
message-list, and the code was not updated.
Fixes#9430#9429 Both of these were due to the same issue, cofx were
wrongly passed to the function resulting in the db being updated but the
fxs being discarded.
There's still a separate issue that might result in messages not being
saved on logout, because of a race condition (if you logout while is
fetching messages, some of the message might not be saved). I will
address that separately as we might be able to just save messages as
they come in status-go, rather then having to pass them to status-react
and back to status-go for saving.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit enables parsing of messages in status-go.
Currently only a few messages are supported in status-protocol-go.
For now we only enable Message types.
Status-react will conditionally use the parsed version if present.
Eventually this can be moved to a separate signal/different structure,
but for the time being is best to validate with the minimum amount of
changes.
The next step would be handle validation and processing of the field in
status-go, so we can skip saving the message from status-react.
This commit should improve performance of receiving messages from a
chat, although haven't had time to validate that.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
The from parameter was left out, tests did not pick it up as they were
not running, probably I have wrongly rebased the code and did not
include the new namespace in runner.cljs
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit does a few things:
==== Ordering of messages ====
Change the ordering of messages from a mixture of timestamp/clock-value to use
only clock-value.
Datemarks are now not used for sorting anymore, which means that the
order of messages is always causally related (not the case before, as we
were breaking this property by sorting by datemark), but datemark
calculation is unreliable (a reply to a message might have a timestamp <
then the message that is replied to).
So for timestamp calculation we
naively group them ignoring "out-of-order timestamp" messages, although
there's much to improve.
It fixes an issue whereby the user would change their time and the
message will be displayed in the past, although it is still possible to
craft a message with a lower clock value and order it in the past
(there's no way we can prevent this to some extent, but there are ways
to mitigate, but outside the scope of this PR).
==== Performance of receiving messages ====
The app would freeze on pulling messages from a mailserver (100 or so).
This is due to the JS Thread being hogged by CPU calculation, coupled
with the fact that we always tried to process messages all in one go.
This strategy can't scale, and given x is big enough (200,300,1000) the
UI will freeze.
Instead, each message is now processed separately, and we leave a gap
between processing each message for the UI to respond to user input
(otherwise the app freezes again).
Pulling messages will be longer overall, but the app will be usuable
while this happen (albeit it might slow down).
Other strategies are possible (calculate off-db and do a big swap,
avoiding many re-renders etc), but this is the reccommended strategy by
re-frame author (Solving the CPU Hog problem), so sounds like a safe
base point.
The underlying data structure for holding messages was also changed, we
used an immutable Red and Black Tree, same as a sorted map for clojure, but we use
a js library as is twice as performing then clojure sorted map.
We also don't sort messages again each time we receive them O(nlogn), but we
insert them in order O(logn).
Other data structures considered but discarded:
1) Plain vector, but performance prepending/insertion in the middle
(both O(n)) were not great, as not really suited for these operations.
2) Linked list, appealing as append/prepend is O(1), while insertion is
O(n). This is probably acceptable as messages tend to come in order
(from the db, so adding N messages is O(n)), or the network (most of
them prepends, or close to the head), while mailserver would not follow this path.
An implementation of a linked list was built, which performed roughtly the
same as a clojure sorted-map (although faster append/prepend), but not
worth the complexity of having our own implementation.
3) Clojure sorted-map, probably the most versatile, performance were
acceptable, but nowhere near the javascript implementation we decided on
4) Priority map, much slower than a sorted map (twice as slow)
5) Mutable sorted map, js implementation, (bintrees), not explored this very much, but from
just a quick benchmark, performance were much worse that clojure
immutable sorted map
Given that each message is now processed separately, saving the chat /
messages is also debounced to avoid spamming status-go with network
requests. This is a temporary measure for now until that's done directly
in status-go, without having to ping-pong with status-react.
Next steps performance wise is to move stuff to status-go, parsing of
transit, validation, which is heavy, at which point we can re-consider
performance and how to handle messages.
Fixes also an issue with the last message in the chat, we were using the
last message in the chat list, which might not necessarely be the last
message the chat has seen, in case messages were not loaded and a more
recent message is the database (say you fetch historical messages for
1-to-1 A, you don't have any messages in 1-to-1 chat B loaded, you receive an
historical message for chat B, it sets it as last message).
Also use clj beans instead of js->clj for type conversion
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit includes a few performance fixes:
1) Pass a string payload instead of an hex encoded string, to avoid
unecessary conversion
2) Don't js->clj on messages, as that's fairly expensive and we can get
away without
3) Don't use `pr-str` `read-string`, rather convert to json
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commits verifies ens names when new messages or contact requests
come through.
A batch of ens names is sent to status-go which will then verifying them
and the result will be passed back in a callback to status-react.
Also temporary skipped test_ens_in_public_chat until we merge the ENS
code (blocked currently by 1.9 upgrade)
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
A user can type in their existing name in the registration flow. Status can
confirm if they own it. After signing a transaction, the user can update the
Whisper ID to their new one.
Instead of using a hardcoded contract for stateofus, the standard `owner`
method is called to find the resolver contract of a ens name.
This allows users to set the pubkey even for ens names that are not
subdomains of stateofus
Signed-off-by: yenda <eric@status.im>
Also:
- add ci/tests/Jenkinsfile.e2e-prs
- remove maybe_later_button click after new onboarding
- update job name for e2e tests
- Fix testrail checklist creationg for nightly builds
Signed-off-by: Jakub Sokołowski <jakub@status.im>
- if multiaccount settings are saved on top of an empty map or nil,
this means something went wrong, the state of the app is unstable,
and actually saving will result in loss of data. It should never
happen, but if it does, throw and error and abort.
- sometimes two fxs are merged when they shouldn't, this is caused by
bugs and should never happen, but if it does, throw an error with arguments
for both effects to help localize the error
Signed-off-by: yenda <eric@status.im>
replace i18n/message-status-label by regular label to avoid repeating
this issue in the future
recover the following labels:
- status-not-sent-click
- status-not-sent-tap
- status-sent
Signed-off-by: yenda <eric@status.im>
- Use community net-info, react-native-webview instead of deprecated react-native classes
- Remove react-native-tcp
- Upgrade react-native libs (react-native-camera, react-native-firebase, react-native-mail, react-native-udp, react-native-webview-bridge)
- Do not include `:react-native-android` module explicitly
- Take advantage of RN AutoLinking
- nix: Update Gradle dependencies
fix#8702
- use ethereum.utils sha3 function to hash passwords before sending them
to status-go
- some native calls take password as one of their params: they now take
hashed-password (still a string, only relevant for status-react caller)
- some native calls take password within a map of rpc params: it now
needs to be hashed with `ethereum.core/sha3`
Signed-off-by: yenda <eric@status.im>
fix#8786
- multiaccount recovery wasn't saving the root key properly
- this resulted in the impossibility to add new accounts in the wallet
Signed-off-by: yenda <eric@status.im>
This commit does a few things:
1) Move messages to status-go
2) Use message-id computed from status-go
3) Remove old replies
Old message id was used for compatibility of replies with older clients.
Given that v1 is breaking, this is not needed anymore and simplifies
moving messages to status-go. No protocol/data-store change is made, to minimize
changes.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Contacts are now in status-go, no migration of contacts is provided so
all contacts will be lost upon installing this build.
I have left the initialization of filters a bit sketchy (we wait that
load-filters is called twice), as the next step will be to avoid calling
load-filters altogether, as now that both contacts & chats are in
status-go, there's no reason to call it from status-react, and can be
called directly from status-go on loading.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit moves chats to status-go.
I have changed the logic to load all chats in one go for simplicity and
while that might have a performance impact, I think it's premature to
optimize this flow as there will be more changes to the login flow.
Also currently this is likely to be slower as we need to wait for the
status-service to be initialized, as well as realm.
No migration is provided as we are past the point of no return, so by
installing this version you will lose your chats.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Currently on some devices there are still some legacy chats(?) that are
one-to-one but don't have a public key as an id (transactor,demo-bot).
We were wrongly sending those to status-go to create filters.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commits changes the behavior to read from a signal instead of
polling each filter.
We receive a signal from status-go every 0.3 seconds, only if new
messages are received. We receive a single signal for all the chats, and
we don't dispatch anymore on every message.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit moves the management of installations to status-go, and
migrates the data from realm.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
All the code has been implemented in statusgo: status-im/status-go#1466
Basically all the whisper filter management is done at that level.
Technical description
On startup we load all chats and send a list of them to status go:
For a public chat: {:chatId "status"}, we create a single filter, based on the name of the chat.
For each contact added by us, each user in a group chat and each one to one chat open, we send:
{:chatId "0x", :oneToOne true}. This will create a chats, to listen to their contact code.
Any previously negotiated topic is also returned.
Once loaded, we create our filters, and upsert the mailserver topics, both of which are solely based on the filters loaded.
In order to remove a chat, we delete/stopwatching first the the filter in status-react and then ask status-go to remove the filter. For a public chat we always remove, for a one-to-one we remove only if the user is not in our contacts, or in a group chat or we have a chat open. Negotiated topics are never removed, as otherwise the other user won't be able to contact us anymore.
On stopping whisper we don't have to ask status-go to remove filters as they are removed automatically.
Some more logic can be pushed in status-go, but that will be in subsequent PRs.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- replace web3-prototype wherever possible
- currently only the money namespace is left
for future refactoring, the ideal solution
would be to use strings for big numbers all
the time and only convert for arithmetic operations
- use json-rpc call to replace trivial web3 calls
Signed-off-by: yenda <eric@status.im>
- upgrade to realm 2.28 to benefit from perf improvements
- remove user-statuses and replace by seen and outgoing-status fields
to get rid of a lot of bloat queries and computations
- remove unused seen message, bottom-infos
- remove unused fields in transport schema
- use objectForPrimaryKey whenever possible instead of get by field
Signed-off-by: yenda <eric@status.im>
Connect to stubs of status-go protocol API, behind the flag. Since status-go isn't updated yet, setting this flag will break the app.
What needs to be tested is no regressions in a normal mode.
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
the defn macro knows register the function as an events under the
keywords provided in the :events key of the attributes map. It also
adds the interceptors provided in the :interceptors map
exemple:
```clojure
(fx/defn hello4
{:doc "this function is useless as well"
:events [:test/valid1 :test/valid2]}
[{:keys [db]} b]
{:db (assoc db :a b) :b (:a db)})
```
Signed-off-by: yenda <eric@status.im>
move utils.ethereum.tokens to ethereum.tokens
move utils.ethereum.abi-spec to ethereum.abi-spec
move utils.ethereum.core to ethereum.core
move utils.ethereum.eip165 to ethereum.eip165
move utils.ethereum.eip55 to ethereum.eip55
move utils.ethereum.eip681 to ethereum.eip681
move utils.ethereum.ens to ethereum.ens
move utils.ethereum.erc721 to ethereum.erc721
move utils.ethereum.mnemonics to ethereum.mnemonics
move utils.ethereum.resolver to ethereum.resolver
move utils.ethereum.macros to ethereum.macros
Signed-off-by: yenda <eric@status.im>
- introduce json-rpc namespace, which provides `call` and `eth-call`,
a generic way of calling a json-rpc method taking care of conversions
and error handling
- remove web3 usage from wallet
- clean up effects, reducing the amount of computations when login in
`wallet-autoconfig-token` is a very expensive call on mainnet
because it checks the balance of every known token.
it is called:
- when wallet is refreshed by pulling
- when user goes on any wallet screen
this PR changes that by:
- calling it only when the wallet is initialized and there is no
visible-token configuration
it only calls update-wallet when a new transaction arrives
- remove the transaction fetching loop entirely to rely only on subscription
for live transactions and token transfer updates
- fetch token transfers history via etherscan API to lift the 100000 blocks
limit on token transfers history
- inbound token transfers are catched via a filter on ethlogs
- outbound token transfers and other transactions are catched by filtering
transaction in current block that have the wallet address as to or from field
- removes fetching of last 100000 blocks of token transfers from
the wallet pull loop
- fetches the last 100000 blocks of token transfers at startup
- replaces pulling by subscriptions to ethlogs for token transfers
- settings are stored in a manifest that is pointed at by the contract
- during login the contract is checked to fetch the settings
Signed-off-by: yenda <eric@status.im>
This PR is part of network incentivisation. It adds a way for a client
to pull nodes from a contract.
This is done by selecting the `eth.contract` fleet. If that is selected
on login it will fetch nodes from a contract and pass them to status-go.
If these can't be fetched, it will default to `eth.beta`.
Currently contract information are hard-coded, but eventually the user
will be able to add their own (probably).
Toggled off in release.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
[#7454] fix add basic copy to public chat empty screen state + chat messages-views intro screens for all chats
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
- support for ipfs only
- provides fns to encode and decode contenthashes as defined in EIP1577
- provides cat fx to retrieve contenthash
Signed-off-by: yenda <eric@status.im>
Sometimes it happens that the expired signal is received while the
there's a new request in flight.
This happens in cases such as:
1) We send a request (A)
2) We get disconnected from the mailserver
3) We connect to a new mailserver
4) We send a request (B)
5) We receive an expired signal for A
In such cases the request should not be retried or counted as a failure.
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
Currently the separate topic was not used, as it's a bit tricky to
coordinate when multiple devices from different versions are present,
with the partitioned topic, probably this optimisation is not necessary
anymore, so removing this for now.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
When we are offline, we don't try to change mailserver, and we don't
show a pop up to the user, as it is not that the mailserver is not
working, we are just offline.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
The denormalized-clock-value was erroneously set to the one of the last
message received. This meant that on chats were the clock-value raced
ahead of the timestamp (#status), a message from the mailserver or a
message from someone with an old clock-value would basically make those
messages be sorted in the past.
The correct behavior is that last-clock-value for a given chat should be
the maximum last clock value ever seen for that chat.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
We keep tokens synchronized across devices, so that the user can notify
us on any paired device.
Currently we record the installation id associated to the fcm-token even
though is not necessary, but it will be once we send device-to-device
messages, in which case we want to notify only those devices.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- remove usage of native checkboxes on android and desktop to ensure proper
design
- rename plain-checkbox to radio button and make sure it is only used where
radio button should be used (one possible choice beyond many)
- update autotest
Signed-off-by: yenda <eric@status.im>
Currently it's very easy for contact details to get out of sync, the
simplest example is:
A & B are contacts.
A changes name.
B receives the updated name.
B re-install the app.
Until A changes name again, B will not see their name, picture and won't
be able to send push notifications.
This PR changes the behavior to publish account informations to contacts
every 24 hrs, to add some redundancy in this cases.
It also publishes a contact code every 12hrs.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- add block/unblock action to user profile
- blocking deletes all messages from user and ignores future messages
- unblocking stops ignoring new messages from user but doesn't recover past ones
[feature] add contact list
[tests] added scroll to BackupRecoveryPhraseButton
[tests] added scroll to public key
Signed-off-by: yenda <eric@status.im>
Currently we use a single topic for discovery.
This provides the best obscurity at the cost of bandwidth, as a message
sent on the discovery topic will be received by any peer.
This PR changes this behavior and start listening on a partitioned
topic.
Each pk will be hashed to a limited number of topics.
Everytime someone is in a conversation with someone from another topic
they will have to listen as well to avoid loosing obscurity, because we
only forward messages that we also advertise in the bloom filter.
The choice for the number of partitions depends on 2 factors:
1) The expected number of users using the network
2) The average number of contacts each user
Any change to the discovery topic will need to be split across 3
releases, to avoid breaking compatibility:
1) Listen to the new and old topic, publish to the old topic
2) Listen to the new and old topic, publish to the new topic
3) Listen to the new topic, publish to the new topic
This is step 1.
This PR enables pairing outside of dev-mode and contact-recovery, which
is useful in the case a new device is added or re-installed.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
As per announcement, we need to switch our Infura project IDs.
> As previously announced, accessing Infura will begin requiring a Project ID generated from the new Infura Dashboard. If you are using Infura and have not yet migrated your project, please take the time to do so now. The first milestone in this transition will be activated next week on January 23, 2019 at 20:00 UTC.
https://blog.infura.io/infura-dashboard-transition-update-c670945a922a
The new project is created with ID `f315575765b14720b32382a61a89341a`
and the API keys are updated.
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
When someone is sending a pfs message to us but did not include our own
device, a pop up is shown propmting the user to connect with the user.
The reason for receiving messages that are not targeting our devices are
various:
1) The account was just recovered (which means it is a new installation
id)
2) More than 3 devices are in use (we only keep max 3 devices in sync)
3) The sender has used an old bundle which does not include the current
device
Eventually we will reduce the likelihood of this scenario happening, but
we can't dismiss it completely.
It's only enabled when PFS is enabled.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit moves group chats to their own topic, based on the randomly
generated chat-id. It falls back on the discovery topic for those peers
who we can't fingerprint the version, for backward compatibility.
Members will now have to explicitly join a group chat to start receiving
messages from it.
Messages are still sent to users who have not joined for backward
compatibility.
Group updates are unaffected.
The bug was introduced in #7055.
`message-type` was stored and used instead of `content-type` which
caused incorrect displaying of the last message preview if the one was a
command.
This commit changes the way how keys are restored:
1. batch of `ssh.addSymKey` requests for all sym keys is sent at once
2. `:status-im.transport.core/sym-keys-added` event is dispatched with
results of all successful `ssh.addSymKey` calls
3. filter is created via `ssh.newMessageFilter`
4. `:shh.callback/filters-added` event is dispatched with all added
filters as a parameter
5. profit
In ideal case only 2 `re-frame` events are dispatched.
In order to get `:last-clock-value` one extra query was executed for
each chat during initialization.
Implementation:
- `:last-clock-value` field was added to `chat` entity
- this field is updated when the message is sent/received
- extra query was removed
Churikova Tetiana
Andrea Maria Piana
Roman Volosovskyi
Serhy
acolytec3
yenda
Andrey Shovkoplyas
Vitaliy Vlasov
Dmitry Novotochinov
tbenr
Pedro Pombeiro
yevh-berdnyk
Julien Eluard
Anton Danchenko
Volodymyr Kozieiev
Jakub Sokołowski
Igor Mandrigin
bitsikka
decanus
mestorlx
kn
Andrea Franz
Igor Mandrigin
Corey