Allow clients to use the profile's data-encryption key as the biometric login credential instead
of the raw password:
- LoginAccount: new `dek` request field (32 hex bytes, validated and normalized to lowercase,
mutually exclusive with password/mnemonic/keycard keys)
- resolveProfileSecret: accepts the client-hashed DEK ("0x" + keccak256), so a stored DEK
credential works through the existing hashed-password paths on the client side
- New ExportProfileDEK endpoint: returns the DEK for a valid credential
Two proxy-backed paths asked for a currency the proxy does not answer in:
- /coins/markets normalizes vs_currency to usd for cache consistency, so a
request for any other currency came back in USD and was then labelled with
the user's currency. convert_currency is the parameter that has an effect
there.
- /simple/price only serves the currencies the proxy holds, so anything else
came back without the requested key and every price silently read 0.
Both now ask for the currency the caller wants through convert_currency and
let the proxy decide where the values come from. The converted currency is
never also listed in vs_currencies, which keeps the request valid whether
the proxy serves provider values for it or converts them.
Only the client pointed at the proxy sends convert_currency; the direct
api.coingecko.com fallback keeps using vs_currency/vs_currencies, which is
what it understands.
The leaderboard endpoints were always fetched without a currency, so the
Market tab showed USD values labelled with whatever currency the user had
selected (status-app#21273). Send convert_currency so the proxy serves the
values in that currency.
The currency comes from the settings DB, seeded at start and followed
through the accounts publisher; the currency a client passes to
FetchMarketTokenPageAsync keeps working as before. Cached values, ETags and
the persisted snapshot all belong to the currency they were fetched in, so a
change drops them and triggers the refresh loops, which push the replacement
data to the client over the same path as any other refresh. With no client
listening the cache is simply left invalidated for the next page request.
Because fetching and storing are not one atomic step, a response carries the
currency it was requested in and is dropped if that no longer matches - a
request in flight across a currency change would otherwise label one
currency's values as another's and refresh the timestamp that decides
whether anything needs fetching.
If the proxy rejects the currency with a 400 the request is retried once
without the conversion, so the tab falls back to USD values rather than
staying empty, and that currency is not asked for again.
Rows persisted before the migration are USD, which is what the proxy served
when no conversion was requested.
doGetRequest built its non-2xx error with fmt.Errorf, leaving a caller that
has to react to one specific status no option but to match on the message.
Return an *HTTPStatusError instead, carrying the code and the body, with an
unchanged Error() string.
The seven network methods on the wallet API were pass-throughs to the
network manager. They now live on the networks service, under the
networks_ namespace.
The three deprecated ones are carried over rather than dropped: the
functional tests still use addEthereumChain to attach the Anvil chain
with a user provider, and getEthereumChains to read it back.
The python client gains a NetworksService and the two call sites move
to it.
params/networkhelper only ever served the network manager and the
default network table, both of which now live in pkg/services/networks.
Eight of its ten functions are unexported now that their callers are in
the same package. OverrideBasicAuth and GetEmbeddedProviders stay
exported: tests in other packages build networks with them.
The two test files become internal tests, since what they cover is no
longer exported.
params/networkdefaults holds the embedded network table and builds the
default network list. It belongs with the service that owns networks,
not in params.
Dissolved into the networks package rather than kept as a subpackage:
BuildDefaultNetworks reads better than defaults.BuildDefaultNetworks,
and the next commit unexports the helpers both halves share.
The network manager lived in internal/rpc and was constructed, started
and stopped by rpc.Client, which reached back into it to route calls by
chain. It is now a service of its own at pkg/services/networks.
StatusNode owns the manager and hands it to rpc.Client through
ClientConfig, so the client depends on ManagerInterface rather than the
concrete type. The service owns the manager lifecycle.
The four live network RPC methods are registered under the networks_
namespace. The wallet_ ones are left in place so nothing breaks before
the app migrates; they are removed at the end of the stack.
BuildDefaultNetworks took requests.WalletSecretsConfig, the raw wire
struct, reaching past the translation that already exists in
buildWalletConfig. It now takes params.WalletConfig, which already
carried six of the seven fields it needs.
PoktAPIKey is added to params.WalletConfig alongside the other provider
keys; it was the only field missing.
This drops networkdefaults' dependency on internal/protocol/requests,
and with it the whole protocol package graph.
Needed for https://github.com/status-im/status-app/issues/21861
Automatically send a contact request to the Status support bot when a profile starts:
- Use a new-user or upgraded-user message based on account creation flow
- Persist request state to prevent repeat sends
- Suppress duplicates on paired devices and when the bot contact already exists
- Add settings migration, pairing handling, startup wiring, and focused tests
Removes the test-only fields (TestsMode, TestEstimationMap, TestBonderFeeMap,
TestApprovalGasEstimation, TestApprovalL1Fee) from ProcessorInputParams and
TestsMode/TestParams (with RouterTestParams and Estimation types) from
requests.RouteInputParams.
Updates the production code accordinglly.
* fix: map envelope.ErrInvalidKEK from the DB-open to incorrect password error
* fix: wrong integrator used
* feat: Status' fee fraction for the LiFi swap added
migrateProfileToDEK now verifies oldKEK before anything is written, by opening the app DB with it
via a new small verifyDBKey helper. A wrong password now returns an error.
The wire format stays password-encrypted keystore files, so pairing works across app
versions (not migrated and DEK migrated profiles) in both directions (sender/receiver):
- sender: migrated profiles re-encrypt keystore files in memory from the DEK to the transfer
password before marshalling
- receiver (account transfer): a brand-new profile adopts a fresh device-local DEK
- receiver (keystore-files transfer): received files are re-encrypted per file to the logged-in
profile's keystore secret
- every failure after the profile keystore directory is created cleans up the profile state
New profiles use the DEK scheme from day one (kdf_iter 3200).
Password change now auto-detects the profile's encryption scheme:
- profile on the DEK scheme, rekey=false → fast path: only the wrapped-DEK file is re-wrapped (no new DEK is generated)
- profile on the DEK scheme, rekey=true → deep rekey: fresh DEK, databases and keystore re-encrypted (new DEK is generated)
- legacy profile → one-time migration to the DEK scheme (full re-encryption, new DEK is generated)
Api changes:
- GetProfileEncryptionInfo endpoint added
- a rekey flag on ChangeDatabasePasswordV2
Part of the Go project layout migration, item 3a: the signing phrase is
no longer a feature, so nothing should generate, store or read it.
pkg/backend/defaults.go buildSigningPhrase and its call
pkg/backend/seed_phrase_dictionary.go the 626-word list it drew from,
which had no other consumer
settings.Settings.SigningPhrase the field and its json tag
settings/database.go the column in INSERT and SELECT
migrations ALTER TABLE settings DROP COLUMN
The column is NOT NULL with no default, so dropping it from the writes
and dropping it from the table have to land together. The migration
follows the pattern of 1779877216_drop_keycard_settings_columns.
refs #7067
Part of the Go project layout migration, item 4.
internal/db/walletdatabase -> internal/db/walletdb, and its one 45-line
file splits along the two jobs it was doing:
open.go DbInitializer, InitializeDB, OpenDB
migrate.go walletCustomSteps, doMigration, MigrateDB
scripts/migration_check.sh listed this migration directory as
"walletdatabase/migrations/sql" and appdatabase's as
"appdatabase/migrations/sql". Neither path has existed since those
packages moved under internal/db/, so the check has been silently
skipping both. Both are corrected here.
refs #7067
Reproducing a reporter's performance problem currently means guessing the
shape of their account: how many messages they hold, how those spread over
chats, how far behind their sync state is. Guessing that cost days on
status-im/status-app#21605.
Add a storagestats service that answers those questions with numbers a
reporter can paste into a public ticket. `storagestats_collect` starts a
background walk and returns immediately; progress arrives as
storage-stats.progress ("N of M", with M known upfront because the table
list is enumerated first) and the finished profile as storage-stats.result.
Nothing is collected unless a caller asks.
The walk is serial on purpose: COUNT(*) on a sqlcipher table is a full
decrypting scan, so running several at once would only starve the process it
is meant to describe. It must never be driven from a client's UI thread.
Privacy is enforced by construction: time appears only as relative day
counts, no per-entity row is ever read (the per-chat histogram query selects
counts and nothing else), and the only strings in the artifact are table
names from our own schema.
Section one carries curated metrics whose names mirror the seeder harness
knobs so a profile maps onto a harness run; section two is a schema-agnostic
table -> {rows, bytes} dump of both databases, insurance against the next
surprise living in a table nobody predicted. A step that fails names itself
in `incomplete` rather than leaving a zero that reads as "this account has
none".
Part of the Go project layout migration, item 31.
Pure move plus import-path rewrite across 687 files. No API or behaviour
change.
The services keep their grouping under pkg/services/<name> rather than
being promoted to pkg/<name>: 27 top-level directories in pkg/ would read
worse than what we have, and the grouping is what makes "an RPC service"
identifiable at a glance.
Paths that follow the move: the logosstorage test target and generate
step, the two wallet token-list tools, the migration-order check (and the
pre-rebase hook symlinked to it), and the storage env helper.
refs #7067
Part of the Go project layout migration, item 27.
Pure move plus import-path rewrite across 502 files. No API or behaviour
change. `internal/` keeps the messaging application logic unimportable
from outside the module, which is what the issue asks for -- status-go is
consumed through the C-bindings in mobile/, not as a Go library.
Things that had to follow the move, beyond the Go imports:
- tools/generate-handlers/template.txt. messenger_handlers.go is
generated, and the template hard-codes the imports it emits, so the
generated file kept importing protocol/common and failed typecheck.
- .gitignore. The ignore rule for that generated file was pinned to the
old path; without moving it, a 1486-line generated file starts being
tracked.
- Makefile: the logosstorage and torrent test targets (both the archive
packages and ./protocol itself), the archive README, migration-protocol.
- scripts/run_unit_tests.sh, which names the protocol package explicitly
to shard its tests.
- scripts/cleanup_generated_files.sh and .golangci.yml.
scripts/migration_check.sh also needed a fix that is not specific to this
move: it validated every file the branch touched under a migration dir
against the timestamp naming rule, and a directory rename makes every
migration in it look newly added. It now excludes renames, so moving a
migration is not mistaken for adding one.
refs #7067
`server` was one Go package doing three unrelated jobs. Each moves to
where it belongs:
server/pairing/ -> services/pairing/
server_media*.go,
handlers*.go, testdata/ -> services/media/
server.go, certs.go,
ips.go, timeout.go,
device.go, listen_*.go,
servertest/ -> internal/httpserver/
MediaServer is renamed to media.Server (and NewMediaServer to
media.NewServer) now that it has a package to be named against.
Deliberately untouched: StatusNode.MediaServer(), which is an accessor
method rather than the type, and the unrelated identifiers that only
share the prefix (MediaServerImageID, MediaServerContactIcon,
MediaServerEnableTLS, ...).
Two identifiers had to be reassigned to make the boundary clean:
- certs.go held both the generic X509/TLS helpers and the media server's
process-global certificate. Split: the generic half stays in
internal/httpserver, generateMediaTLSCert and PublicMediaTLSCert move
to services/media.
- HandlerPatternMap was declared in handlers.go but is a plain HTTP type
that server.go depends on; it moves to internal/httpserver.
The media server URL tests moved with the type, and reached three
unexported Server fields they could touch while everything shared a
package. internal/httpserver now exposes ListeningAddr() and CachedPort()
(both reasonable API) plus a clearly-marked SetURLStateForTest.
Nothing else crossed the boundary, and the split is one-directional:
services/media and services/pairing import internal/httpserver, never the
reverse.
refs #7067
`common` was a grab-bag with no domain: the issue's own preamble names
it as the kind of package that must not exist. Every symbol moves to the
package that owns it, and the directory is deleted.
common/dbsetup -> internal/db/dbsetup
common/devices.go -> internal/platform
common/pausable*.go -> internal/pausable
LogOnPanic -> internal/panics
TruncateWithDot(N) -> internal/logutils
RecoverKey, ValidateDisplayName, display-name errors -> protocol/common
IpfsGatewayURL -> internal/ipfs.GatewayURL
Archives/TorrentTorrentsRelativePath, MainnetEthereumNetworkURL -> params
StatusService -> pkg/backend/node
ErrBigIntSetFromString -> services/wallet
IsNil, Ptr -> inlined at their call sites
IsENSName -> deleted, it had no callers
Notes:
- LogOnPanic gets its own package rather than living in logutils. It
reports to Sentry, and logutils is imported by nearly everything: put
the guard in logutils and the Sentry SDK lands in every dependency
graph in the tree (213 -> 250 packages). internal/panics imports
logutils and sentry, which is the direction root `common` had.
- TruncateWithDot is log redaction, not string formatting: every one of
its 121 call sites is inside a log or error message, so it belongs
next to the logger.
- Moving RecoverKey and ValidateDisplayName into protocol/common removes
the common -> protocol layering inversion; all their callers were
already inside protocol/.
- Makefile lint-panics target follows LogOnPanic to its new path.
refs #7067
Part of the Go project layout migration. Pure move plus import-path
rewrite; the public API is unchanged.
The "emit signals without global functions" redesign is a behavioural
change that touches the C-binding callback and is deliberately left out
of this PR.
refs #7067
Fixes#7620
Return ErrNoStorenodesReachable when a history query arrives before
StoreClient initialization, instead of dereferencing a nil client.
Publish the fully configured StoreClient under a mutex and synchronize
query and active-storenode reads to prevent a startup race.
Add a regression test covering StoreQuery before Waku Start.
Part of https://github.com/status-im/status-app/issues/21544
Use a non-blocking, quit-aware send for new hash ratchet key notifications.
Drop and warn when the subscriber queue is full, and skip notifications for
empty key payloads.
Add regressions for full notification queues and empty key payloads.
Session archives created by rotateLogFileForNewSession used the timestamp format "2006-01-02T15-04-05Z",
which lumberjack's cleanup cannot parse (its backupTimeFormat is "2006-01-02T15-04-05.000"). Because of that
archives were never counted against MaxBackups and accumulated forever.
Changes:
- rotateLogFileForNewSession now archives with lumberjack's exact backupTimeFormat
- renameLegacySessionArchives renames existing "...Z" archives to the new format on logger init,
making old files removable too
- if not set the MaxBackups becomes DefaultLogMaxBackups (10)
* chore: drop redundant numeric import aliases
Refactoring left behind import aliases like `datasync2`/`types3` that
just repeat the package name. Remove them where the plain package name is
unambiguous in the file, and drop three duplicate imports of the same path.
* chore: name colliding type imports after repo convention
Where two packages named `types` (or `rpc`) meet in one file an alias is
unavoidable, so use the descriptive names already dominant in the tree
(cryptotypes, messagingtypes, wakutypes, accsmanagementtypes, wsdktypes,
noderpc) instead of types2/types3/rpc2, and spell the messaging/waku
import wakuv2 everywhere (was wakuv/wakuv2/wakuv3).
* chore: rename package wakuv2 to waku
The package in pkg/messaging/waku still declared itself `wakuv2`, which
forced every import site to carry an alias (goimports re-adds one when the
package name differs from its directory). Rename the package so the import
can stand as-is, and rename the local `waku` variable in transport_test.go
to `wakuNode` to free up the name.
* chore: unalias protocol/contacts import in protocol tests and backup
`contacts2` was only needed because local variables took the package name.
Rename those to what they hold — syncContacts for the sync messages built in
backupContacts, addedContacts/allContacts in the contact request and
verification tests — and import the package as-is.
processMessage legitimately returns (nil, nil) when a message is still
segmented-incomplete or gets re-queued because its hash ratchet key has
not arrived yet. processQueuedHashRatchetMessages dereferenced that nil
response (the shadowed r), crashing the whole app with SIGSEGV. Skip
such messages and leave them in the queue.
Red on the parent commit: the replay dereferenced the nil response and
crashed. Green with the guard: the stray message stays queued under its
own key and the rest of the replay completes.
Fixes#7568
- bound automatic history reconciliation to observed unreliable-delivery windows
- introduce typed reconciliation events carrying fixed `From` and `To` bounds
- retain, retry, and coalesce pending windows while preserving disjoint outages
- prevent stable online periods from being queried during later reconciliation
- use `mailserver_topics.last_request` as a monotonic “known complete through” cursor
- checkpoint initialized topic cursors during reliable full-node connectivity
- schedule bounded reconciliation for offline recovery, sleep/wake, and pause/resume
- remove duplicate startup fetching and retain one cursor-based startup catch-up
- wake retained work when mobile-network syncing is enabled
- bound newest-community-description queries to the reconciliation window
- preserve existing behavior for initial topic history, manual requests, and archive backfills
Reliability.Stop() destroyed the SDS reliability manager while Start() only
rebuilt the mvds datasync node, so the first offline->online transition (driven
by Core.connectionChanged) left SDS nil for the rest of the process and every
subsequent message — live or fetched from a store node — arrived still
SDS-wrapped, failed to decode at the application layer and surfaced as type
UNKNOWN. Stop() now tears down only the datasync node and preserves SDS, whose bloom filter and causal history are exactly the state needed to detect what was missed while offline; a new Close() releases it on shutdown, and Start()
rebuilds it if it is ever missing. Two related bugs are fixed alongside:
UnwrapPayloadFromSDS now returns ErrSDSManagerUnavailable when the manager is gone instead of silently passing the wrapped payload through (it still passes through, error-free, when a payload is simply not SDS-wrapped), and the
processor propagates that error so the envelope is retried rather than
confirmed as processed. sdsManager is stored in an atomic.Pointer, since the
hot path read it without holding the lock that Start/Stop write under.
Part of https://github.com/status-im/status-app/issues/21462
- Open existing app and wallet SQLCipher databases concurrently, while preserving sequential initialization for new or legacy databases.
- Speed up account selection by preloading the profile keypair and decrypting only the chat private key instead of the full extended key.
- Defer token manager startup until after login completes and run it asynchronously outside the critical startup path.
- Load cached leaderboard market data asynchronously, waiting only when the data is accessed or the service stops.
Before: about 2.15 s for the backend login request
After: about 0.47–0.50 s
Improvement: roughly 1.65–1.70 s saved
Relative reduction: about 77–78%
Speed multiplier: approximately 4.3–4.5× faster
Part of https://github.com/status-im/status-app/issues/21598
Updated outgoing message status handling so `sent` is confirmed immediately after successful Waku publishing, eliminating store-node confirmation requests and retaining only MVDS ACKs (1:1/private groups) and SDS `message_sent` callbacks (communities) for `delivered`. Community SDS IDs are now tracked as internal aliases of the UI message ID, preventing alias messages from appearing in the UI while translating SDS delivery callbacks to the existing delivered signal.
Fixes#7363
The SDS library can detect some messages are missing by accident. Either because it didn't process them yet or just because the app was restarted.
This change checks the cache before fetching to make sure we are not fetching messages we already know.
Fixes#7363
Enables the SDS wrapping flag.
Sets up the handler that wraps SDS messages with retrieval hints. Those hints are the envelope IDs of the messages that were sent and received
Sets up the unwrapping and fetching when there are missed messages detected.
Adds a new function that enables fetching per envelope ID instead than per topic.
saledjenic