This commit does a few things:
==== Ordering of messages ====
Change the ordering of messages from a mixture of timestamp/clock-value to use
only clock-value.
Datemarks are now not used for sorting anymore, which means that the
order of messages is always causally related (not the case before, as we
were breaking this property by sorting by datemark), but datemark
calculation is unreliable (a reply to a message might have a timestamp <
then the message that is replied to).
So for timestamp calculation we
naively group them ignoring "out-of-order timestamp" messages, although
there's much to improve.
It fixes an issue whereby the user would change their time and the
message will be displayed in the past, although it is still possible to
craft a message with a lower clock value and order it in the past
(there's no way we can prevent this to some extent, but there are ways
to mitigate, but outside the scope of this PR).
==== Performance of receiving messages ====
The app would freeze on pulling messages from a mailserver (100 or so).
This is due to the JS Thread being hogged by CPU calculation, coupled
with the fact that we always tried to process messages all in one go.
This strategy can't scale, and given x is big enough (200,300,1000) the
UI will freeze.
Instead, each message is now processed separately, and we leave a gap
between processing each message for the UI to respond to user input
(otherwise the app freezes again).
Pulling messages will be longer overall, but the app will be usuable
while this happen (albeit it might slow down).
Other strategies are possible (calculate off-db and do a big swap,
avoiding many re-renders etc), but this is the reccommended strategy by
re-frame author (Solving the CPU Hog problem), so sounds like a safe
base point.
The underlying data structure for holding messages was also changed, we
used an immutable Red and Black Tree, same as a sorted map for clojure, but we use
a js library as is twice as performing then clojure sorted map.
We also don't sort messages again each time we receive them O(nlogn), but we
insert them in order O(logn).
Other data structures considered but discarded:
1) Plain vector, but performance prepending/insertion in the middle
(both O(n)) were not great, as not really suited for these operations.
2) Linked list, appealing as append/prepend is O(1), while insertion is
O(n). This is probably acceptable as messages tend to come in order
(from the db, so adding N messages is O(n)), or the network (most of
them prepends, or close to the head), while mailserver would not follow this path.
An implementation of a linked list was built, which performed roughtly the
same as a clojure sorted-map (although faster append/prepend), but not
worth the complexity of having our own implementation.
3) Clojure sorted-map, probably the most versatile, performance were
acceptable, but nowhere near the javascript implementation we decided on
4) Priority map, much slower than a sorted map (twice as slow)
5) Mutable sorted map, js implementation, (bintrees), not explored this very much, but from
just a quick benchmark, performance were much worse that clojure
immutable sorted map
Given that each message is now processed separately, saving the chat /
messages is also debounced to avoid spamming status-go with network
requests. This is a temporary measure for now until that's done directly
in status-go, without having to ping-pong with status-react.
Next steps performance wise is to move stuff to status-go, parsing of
transit, validation, which is heavy, at which point we can re-consider
performance and how to handle messages.
Fixes also an issue with the last message in the chat, we were using the
last message in the chat list, which might not necessarely be the last
message the chat has seen, in case messages were not loaded and a more
recent message is the database (say you fetch historical messages for
1-to-1 A, you don't have any messages in 1-to-1 chat B loaded, you receive an
historical message for chat B, it sets it as last message).
Also use clj beans instead of js->clj for type conversion
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit includes a few performance fixes:
1) Pass a string payload instead of an hex encoded string, to avoid
unecessary conversion
2) Don't js->clj on messages, as that's fairly expensive and we can get
away without
3) Don't use `pr-str` `read-string`, rather convert to json
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commits verifies ens names when new messages or contact requests
come through.
A batch of ens names is sent to status-go which will then verifying them
and the result will be passed back in a callback to status-react.
Also temporary skipped test_ens_in_public_chat until we merge the ENS
code (blocked currently by 1.9 upgrade)
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
A user can type in their existing name in the registration flow. Status can
confirm if they own it. After signing a transaction, the user can update the
Whisper ID to their new one.
Instead of using a hardcoded contract for stateofus, the standard `owner`
method is called to find the resolver contract of a ens name.
This allows users to set the pubkey even for ens names that are not
subdomains of stateofus
Signed-off-by: yenda <eric@status.im>
currently navigation-reset only works properly if you reset to a simple route
this fix allows us to use navigation reset with more complex routes, by
ensuring that the navigation stack is populated properly so that navigate
back doesn't end up emptying the stack.
this is temporary as the proper way to do navigation in general would be
to get rid of view id and navigation-stack entirely, since it is a duplication
of the state of react-navigation
Signed-off-by: yenda <eric@status.im>
master key
When creating the account we store as well the path specified in eip1581
https://github.com/ethereum/EIPs/blob/master/EIPS/eip-1581.md ,
`m / 43' / 60' / 1581'`.
The reason for doing so is that eventually we might want to derive an
encryption key from it, which would require the user to re-enter their
seed phrase if we would not store this.
This commit changes the behavior not to store the master key, and
instead store `m /44'/60' /0'/0`, from which wallets are now derived.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- if multiaccount settings are saved on top of an empty map or nil,
this means something went wrong, the state of the app is unstable,
and actually saving will result in loss of data. It should never
happen, but if it does, throw and error and abort.
- sometimes two fxs are merged when they shouldn't, this is caused by
bugs and should never happen, but if it does, throw an error with arguments
for both effects to help localize the error
Signed-off-by: yenda <eric@status.im>
- renamed the macros def and defn so that they are now used with aliased
namespace `styles/def` and `styles/defn` to force user to use aliased require
instead of require-macro and refer
- this makes sure the cljs file is required which includes the require for
platform ns needed after macroexpension
Signed-off-by: yenda <eric@status.im>
replace i18n/message-status-label by regular label to avoid repeating
this issue in the future
recover the following labels:
- status-not-sent-click
- status-not-sent-tap
- status-sent
Signed-off-by: yenda <eric@status.im>
safe merge was using way too much inefficient code for such an important
function
it is rewritten using a reduce. the performance improvement is 10 times
and should really show up when adding messages
in repl session the new merge was much slower on the error case of merging
fx with common keys but it must never happen in production as it means
the app is broken
status-im.utils.fx> (time (dotimes [x 100] (fast-merge {:a 1 :b 2 :filters/load-filters [{:a 1 :b 2}]} {:c 3 :filters/load-filters [{:d 1 :b x}]})))
"Elapsed time: 19.000000 msecs"
nil
status-im.utils.fx> (time (dotimes [x 100] (safe-merge {:a 1 :b 2 :filters/load-filters [{:a 1 :b 2}]} {:c 3 :filters/load-filters [{:d 1 :b x}]})))
"Elapsed time: 183.000000 msecs"
status-im.utils.fx> (time (dotimes [x 100] (fast-merge {:a 1 :c 2 :filters/load-filters [{:a 1 :b 2}]} {:c 3 :filters/load-filters [{:d 1 :b x}]})))
"Elapsed time: 2224.000000 msecs"
- Use community net-info, react-native-webview instead of deprecated react-native classes
- Remove react-native-tcp
- Upgrade react-native libs (react-native-camera, react-native-firebase, react-native-mail, react-native-udp, react-native-webview-bridge)
- Do not include `:react-native-android` module explicitly
- Take advantage of RN AutoLinking
- nix: Update Gradle dependencies
fix#8702
- use ethereum.utils sha3 function to hash passwords before sending them
to status-go
- some native calls take password as one of their params: they now take
hashed-password (still a string, only relevant for status-react caller)
- some native calls take password within a map of rpc params: it now
needs to be hashed with `ethereum.core/sha3`
Signed-off-by: yenda <eric@status.im>
- remove impl/modules.cljs namespace because it was needlessly redundant
- remove `create-account` unused call
- add documentation for some calls
- only keep `(status)` check for relevant calls that are failing tests
because they are evaluated
We use the timestamp of the last message in the chat preview.
In case there's no message, the old timestamp will be displayed (last
time the chat has been updated).
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Fixes#8635 by adding VERSION and BUILD_NUMBER files in the correct nix
template and updating bash script in order to prevent it from failing,
due to the git repository being not initialized in the nix environment.
Move scripts/build_no.sh and scripts/gen_build_no.sh to
scripts/version/build_no.sh to prevent Nix from rebuilding when
unrelated scripts are touched.
Signed-off-by: Andrey Shovkoplyas <motor4ik@gmail.com>
when unlocking first blocked contact before relogin, an error occurs
this is because the first blocked contact is added to a list and you can't
disj an element from a list
this makes sure the first blocked contact is put into a set
Signed-off-by: Andrey Shovkoplyas <motor4ik@gmail.com>
fix#8786
- multiaccount recovery wasn't saving the root key properly
- this resulted in the impossibility to add new accounts in the wallet
Signed-off-by: yenda <eric@status.im>
This commit does a few things:
1) Move messages to status-go
2) Use message-id computed from status-go
3) Remove old replies
Old message id was used for compatibility of replies with older clients.
Given that v1 is breaking, this is not needed anymore and simplifies
moving messages to status-go. No protocol/data-store change is made, to minimize
changes.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit adds datasync confirmations and enables device-to-device for
all the communications, as that's what we will go with v1.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Contacts are now in status-go, no migration of contacts is provided so
all contacts will be lost upon installing this build.
I have left the initialization of filters a bit sketchy (we wait that
load-filters is called twice), as the next step will be to avoid calling
load-filters altogether, as now that both contacts & chats are in
status-go, there's no reason to call it from status-react, and can be
called directly from status-go on loading.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit moves chats to status-go.
I have changed the logic to load all chats in one go for simplicity and
while that might have a performance impact, I think it's premature to
optimize this flow as there will be more changes to the login flow.
Also currently this is likely to be slower as we need to wait for the
status-service to be initialized, as well as realm.
No migration is provided as we are past the point of no return, so by
installing this version you will lose your chats.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
Currently on some devices there are still some legacy chats(?) that are
one-to-one but don't have a public key as an id (transactor,demo-bot).
We were wrongly sending those to status-go to create filters.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
In preparation for v1 this commits adds a few options so we can get
start debugging the protocol for v1.
This options are:
1) Datasync: If enabled it will send datasync messages
2) V1Messages: If enabled it will send v1 messages (just adding a
signature to the message)
3) Disable discovery topic: If enabled it will stop listening/publishing
on the discovery topic. You will be able to receive messages only from
clients who have this enabled as well.
If any of this option is on, it will only be compatitle with builds >=
this one. A logout is required for any change to take effect.
All this options will be removed before v1, they are there just to make
it easier for us to test and find potential issues.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
- pending transaction was using hash so mined transaction was using it's id
to have a unique id, the fix adds a rule when the existing transaction by hash
is in :pending state, the mined transaction overwrites it
Signed-off-by: yenda <eric@status.im>
This commits changes the behavior to read from a signal instead of
polling each filter.
We receive a signal from status-go every 0.3 seconds, only if new
messages are received. We receive a single signal for all the chats, and
we don't dispatch anymore on every message.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
This commit moves the management of installations to status-go, and
migrates the data from realm.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
All the code has been implemented in statusgo: status-im/status-go#1466
Basically all the whisper filter management is done at that level.
Technical description
On startup we load all chats and send a list of them to status go:
For a public chat: {:chatId "status"}, we create a single filter, based on the name of the chat.
For each contact added by us, each user in a group chat and each one to one chat open, we send:
{:chatId "0x", :oneToOne true}. This will create a chats, to listen to their contact code.
Any previously negotiated topic is also returned.
Once loaded, we create our filters, and upsert the mailserver topics, both of which are solely based on the filters loaded.
In order to remove a chat, we delete/stopwatching first the the filter in status-react and then ask status-go to remove the filter. For a public chat we always remove, for a one-to-one we remove only if the user is not in our contacts, or in a group chat or we have a chat open. Negotiated topics are never removed, as otherwise the other user won't be able to contact us anymore.
On stopping whisper we don't have to ask status-go to remove filters as they are removed automatically.
Some more logic can be pushed in status-go, but that will be in subsequent PRs.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
All resources loaded by slurp are moved to status-modules/resources dir
in release builds and are loaded only by demand instead of being bundled into
index.*.js.
- replace web3-prototype wherever possible
- currently only the money namespace is left
for future refactoring, the ideal solution
would be to use strings for big numbers all
the time and only convert for arithmetic operations
- use json-rpc call to replace trivial web3 calls
Signed-off-by: yenda <eric@status.im>
- remove font-weight and font-style on mobile because otherwise android
falls back to default font and doesn't use the font-family
- since nested text inherits from parent, only put the needed styling
in nested text style
- use parens instead of brackets to resolve components
- upgrade to realm 2.28 to benefit from perf improvements
- remove user-statuses and replace by seen and outgoing-status fields
to get rid of a lot of bloat queries and computations
- remove unused seen message, bottom-infos
- remove unused fields in transport schema
- use objectForPrimaryKey whenever possible instead of get by field
Signed-off-by: yenda <eric@status.im>
This concerns:
- touchable-opacity
- animated-view
- scroll-view
Tip for rebasing:
search in project for `scroll-view)` `animated-view)` and `touchable-opacity)`
Signed-off-by: yenda <eric@status.im>
- allows transaction flow started by an internal feature to show
the right asset in the overview and in transaction messages
(required for tribute to talk)
Signed-off-by: yenda <eric@status.im>
Connect to stubs of status-go protocol API, behind the flag. Since status-go isn't updated yet, setting this flag will break the app.
What needs to be tested is no regressions in a normal mode.
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
the defn macro knows register the function as an events under the
keywords provided in the :events key of the attributes map. It also
adds the interceptors provided in the :interceptors map
exemple:
```clojure
(fx/defn hello4
{:doc "this function is useless as well"
:events [:test/valid1 :test/valid2]}
[{:keys [db]} b]
{:db (assoc db :a b) :b (:a db)})
```
Signed-off-by: yenda <eric@status.im>
move utils.ethereum.tokens to ethereum.tokens
move utils.ethereum.abi-spec to ethereum.abi-spec
move utils.ethereum.core to ethereum.core
move utils.ethereum.eip165 to ethereum.eip165
move utils.ethereum.eip55 to ethereum.eip55
move utils.ethereum.eip681 to ethereum.eip681
move utils.ethereum.ens to ethereum.ens
move utils.ethereum.erc721 to ethereum.erc721
move utils.ethereum.mnemonics to ethereum.mnemonics
move utils.ethereum.resolver to ethereum.resolver
move utils.ethereum.macros to ethereum.macros
Signed-off-by: yenda <eric@status.im>
in many places in the codebase, we are doing various destructuring
and function calls to get the normalized current address and the chain
keyword for the current network
this PR replace all usages by utility functions introduced recently
Signed-off-by: yenda <eric@status.im>
- temporary fix before visible assets are fixed
- the problem is that you can't hide an asset with the way it currently
works, it will come back every time as long as there is a balance for it
Signed-off-by: yenda <eric@status.im>
- introduce json-rpc namespace, which provides `call` and `eth-call`,
a generic way of calling a json-rpc method taking care of conversions
and error handling
- remove web3 usage from wallet
- clean up effects, reducing the amount of computations when login in
`wallet-autoconfig-token` is a very expensive call on mainnet
because it checks the balance of every known token.
it is called:
- when wallet is refreshed by pulling
- when user goes on any wallet screen
this PR changes that by:
- calling it only when the wallet is initialized and there is no
visible-token configuration
it only calls update-wallet when a new transaction arrives
- remove the transaction fetching loop entirely to rely only on subscription
for live transactions and token transfer updates
- fetch token transfers history via etherscan API to lift the 100000 blocks
limit on token transfers history
- inbound token transfers are catched via a filter on ethlogs
- outbound token transfers and other transactions are catched by filtering
transaction in current block that have the wallet address as to or from field
- removes fetching of last 100000 blocks of token transfers from
the wallet pull loop
- fetches the last 100000 blocks of token transfers at startup
- replaces pulling by subscriptions to ethlogs for token transfers
For associative collections, using reduce-kv should
give a small performance gain over reduce as there is
no allocation of tuples for key/value pairs
Signed-off-by: yenda <eric@status.im>
- use signal subscriptions to get latest block signals
- compute confirmations in transaction details from current block
dynamically
Signed-off-by: yenda <eric@status.im>
- settings are stored in a manifest that is pointed at by the contract
- during login the contract is checked to fetch the settings
Signed-off-by: yenda <eric@status.im>
- when login in after a different account using a different network
the new account inherited the network from the previous one
- this fixes it by using the account network when initializing app-db
instead of reusing current network
Signed-off-by: yenda <eric@status.im>
- previously wallet transaction fetching was quite hazardous
- most of the time the `:wallet/all-tokens` map was nil so no token
was recognized
- the query for transfers was made accross 100000 blocks, now it
only checks that the first time, then from latest block checked minus
12 to follow progress of unconfirmed transactions
Signed-off-by: yenda <eric@status.im>
- change `ethereum/get-transaction` to use call-private-rpc instead
of web3 object
- implement a version for erc20 tokens SNT and STT that provides enriches
result of get-transaction with `:recipient` and `:snt-amount`
Signed-off-by: yenda <eric@status.im>
- turn :send-transaction-message event into an fx producing function
- allow transaction with defined on-result event to also send transaction
message
Signed-off-by: yenda <eric@status.im>
This PR is part of network incentivisation. It adds a way for a client
to pull nodes from a contract.
This is done by selecting the `eth.contract` fleet. If that is selected
on login it will fetch nodes from a contract and pass them to status-go.
If these can't be fetched, it will default to `eth.beta`.
Currently contract information are hard-coded, but eventually the user
will be able to add their own (probably).
Toggled off in release.
Signed-off-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
[#7454] fix add basic copy to public chat empty screen state + chat messages-views intro screens for all chats
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
* `open-sign-transaction-flow` cleaner version of
`open-modal-wallet-for-transaction`
* The contract namespace provides the `call` effect:
- takes contract identifier from contracts, method, params and callback
- select the correct contract address based on current network
- encode params, decodes results
- start wallet transaction flow for write calls
Signed-off-by: yenda <eric@status.im>
- support for ipfs only
- provides fns to encode and decode contenthashes as defined in EIP1577
- provides cat fx to retrieve contenthash
Signed-off-by: yenda <eric@status.im>
- remove web3 parameter for ethereum/call
- remove first parameter for callback of ethereum/call which
was always nil
Signed-off-by: yenda <eric@status.im>
In some cases (say the clock of the device has drifted) specifying a `to` parameter
is decrimental and might result in missing messages.
We have changed the mailservers to default `to` if not present to `now`,
which should gives us better guarantees.
We also removed the 24 hours limit as now all the requests will be
paginated according to the `limit` parameter.
Signed-off-by: Igor Mandrigin <i@mandrigin.ru>
Andrey Shovkoplyas
William Mruzek
Andrea Maria Piana
Roman Volosovskyi
acolytec3
yenda
tbenr
Serhy
Vitaliy Vlasov
Marcus R. Brown
Dmitry Novotochinov
Alex Good
Volodymyr Kozieiev
Pedro Pombeiro
bitsikka
jiachen
b-m-f
MajorTomSec
Julien Eluard
Rikku
Igor Mandrigin
Alexandre Magno
Anton Danchenko
Dmitry
Edvard
Churikova Tetiana
Goran Jovic
Brandon Wissmann
Kris Calabio
Emilio Silva Schlenker
Jakub Sokołowski
etherman
Speedy Fixer