Part of the Go project layout migration, item 4.
internal/db/walletdatabase -> internal/db/walletdb, and its one 45-line
file splits along the two jobs it was doing:
open.go DbInitializer, InitializeDB, OpenDB
migrate.go walletCustomSteps, doMigration, MigrateDB
scripts/migration_check.sh listed this migration directory as
"walletdatabase/migrations/sql" and appdatabase's as
"appdatabase/migrations/sql". Neither path has existed since those
packages moved under internal/db/, so the check has been silently
skipping both. Both are corrected here.
refs #7067
Part of the Go project layout migration, item 31.
Pure move plus import-path rewrite across 687 files. No API or behaviour
change.
The services keep their grouping under pkg/services/<name> rather than
being promoted to pkg/<name>: 27 top-level directories in pkg/ would read
worse than what we have, and the grouping is what makes "an RPC service"
identifiable at a glance.
Paths that follow the move: the logosstorage test target and generate
step, the two wallet token-list tools, the migration-order check (and the
pre-rebase hook symlinked to it), and the storage env helper.
refs #7067
Part of the Go project layout migration, item 27.
Pure move plus import-path rewrite across 502 files. No API or behaviour
change. `internal/` keeps the messaging application logic unimportable
from outside the module, which is what the issue asks for -- status-go is
consumed through the C-bindings in mobile/, not as a Go library.
Things that had to follow the move, beyond the Go imports:
- tools/generate-handlers/template.txt. messenger_handlers.go is
generated, and the template hard-codes the imports it emits, so the
generated file kept importing protocol/common and failed typecheck.
- .gitignore. The ignore rule for that generated file was pinned to the
old path; without moving it, a 1486-line generated file starts being
tracked.
- Makefile: the logosstorage and torrent test targets (both the archive
packages and ./protocol itself), the archive README, migration-protocol.
- scripts/run_unit_tests.sh, which names the protocol package explicitly
to shard its tests.
- scripts/cleanup_generated_files.sh and .golangci.yml.
scripts/migration_check.sh also needed a fix that is not specific to this
move: it validated every file the branch touched under a migration dir
against the timestamp naming rule, and a directory rename makes every
migration in it look newly added. It now excludes renames, so moving a
migration is not mistaken for adding one.
refs #7067
tests-functional/ -> test/functional/
tests-unit-network/ -> test/unit-network/
No Go code changes beyond the two package paths. Every reference follows:
Makefile targets, the pytest-lint and reliability workflows, the three
Jenkinsfiles, the root Dockerfile, pyrightconfig.json, .gitignore, the
benchmark and functional-test scripts, and the docs.
All paths inside the moved directories are self-contained (docker compose
uses `context: .`, the Python helpers resolve from __file__), so the extra
level of nesting does not reach outside.
refs #7067
* ci(tests): report coverage to Codecov on PRs
codecov upload-process relied on codecov-cli auto-detection, which doesn't
associate uploads with the pull request (and on merge-commit heads uploads to
the wrong commit), so Codecov posts no status checks on PRs while develop keeps
reporting.
Pass commit/branch/PR explicitly from Jenkins env (GIT_COMMIT, params.BRANCH,
utils.changeId()). No change to develop reporting (no PR -> --pr omitted).
* ci(tests): drop redundant null guards in codecov upload
Keep the -n guards (they omit --pr on develop where there is no PR, and keep local runs working when the vars are unset).
---------
Co-authored-by: Egor Rachkovskii <egorrachkovskii@status.im>
* chore: adds setting file for NeoVim
* feat: wires up LogosStorageConfig db setup
* feat: adds implementation of ArchiveManagerLogosStorage and wires it up
* feat: wire LogosStorage history archive runtime control
* feat: expose history archive index completion signal
* feat: expose LogosStorage debug and connect APIs
* feat: add history archive timing knobs
* chore: align functional archive build toggles
* chore: refactors token permissions tests
* feat: wire LogosStorage archive APIs, request plumbing, and functional test
* fix: use UniversalChatID() for archive link distribution
* fix: functional archive tests
* feat: respecting online status in history archive task
* feat: more control over "ratchetNotFoundDelay"
* test: run storage-related functional tests only when USE_LOGOS_STORAGE is true
* test: adds LogosStorage backend tests
* build: updates testing on CI
* build: updates linting on CI
* fix: make sure archive_manager_torrent_test has use_torrent guard
* fix: formatting
* chore: updates nvim settings to include CGO paths
* chore: removes redundant "special disabled" files
* fix: Python linting issues
* build: use system Nim for libstorage native and Docker builds
* fix: initialize LogosStorage node config in DefaultNodeConfig
* fix: rebase
* fix: final review
* build: cleans up the vars in test storage and torrent targets in Makefile
* fix: stop unseeding before calling CreateHistoryArchiveFromDB (logos)
* test: cleans up some noise when closing websockets in functional tests
* test: further increase test coverage
* fix: linting
* chore: update local project nvim config
* chore: refactors ArchiveManagerLogosStorage
* chore: updates CONTRIBUTING.md
* build: temporarily add mc2 convenience scripts
* build: reset shared dependency repos before checkout in Makefile
Adds `git reset --hard` before `git checkout` in the `clone-nim-sds`
and `clone-storage` Makefile targets. Prevents CI failures when the
shared `../nim-sds` or `../logos-storage-nim` directories contain
unexpected local modifications from previous builds.
* fix: use shorter waku message retention policy only for LogosStorage tests
* fix: rename COMMUNITY_IMPORTING_HISTORY_ARCHIVE_MESSAGES_FINISHED to COMMUNITY_HISTORY_ARCHIVES_DOWNLOAD_AND_IMPORT_FINISHED
* build: move STORAGE and TORRENT envs close to their respective targets
* build: more robust cloning target for sds and storage
* fix(waku): set filter full-node rate limit to match the fleet (30/min)
status-go enables the go-waku filter server on full nodes but only set
WithMaxSubscribers, leaving the rate limiter at go-waku's default of
1 request/second, burst 1 — while the lightpush server right beside it
got an explicit limiter. A light client joining a community fans out to
many per-content-topic FilterSubscribeRequests at once, so a 1/s+burst-1
server rejects them with 429 ("filter request rejected due rate limit
exceeded") for ~60s+, longer than the functional tests' 60s wait, so
pushed community messages are dropped server-side ("no subscribed peers
found") and the light receiver times out (flaky full-light/light-light
compat cells).
Set the filter full-node limiter to 30 requests/minute per peer (burst
30), matching the logos-delivery (nwaku) fleet default
(FilterDefaultPerPeerRateLimit = (30, 1.minutes)).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(functional): serve community shards 128 & 256 in the waku fleet
Old peers (v10.33.x) subscribe community channel filters on shards
rs/16/128 and rs/16/256, but the test fleet only served shards 32 and 64.
Those subscribes fail with 'no suitable peers found' and churn the light
client's filter manager, collaterally destabilizing the shard-32
subscription that actually carries the community message -> the light
receiver misses it and times out (flaky full-light / light-light compat
cells against v10.33.2).
Subscribe the fleet (boot-1 + store) to shards 128 and 256 as well, and
bump --num-shards-in-network to 257 so those shards validate. This is a
fleet-only change; the value is not propagated to the status-go backends
(the fleet-config scanner only emits clusterId + node ENRs), so it does
not affect how any backend derives shards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(functional): use nwaku-pr:3939 fleet (PX stale-ENR prune fix)
The light-client compat flakiness is driven by peer-exchange pollution:
the shared test fleet's PX server hands each light client the ENRs of
every other parallel cell's backends (incl. torn-down ones). The light
client churns dialing those dead peers ('all dials failed') and its
filter peer-selection intermittently can't find the connected fleet node
for the community shard ('no suitable peers found') -> missed message ->
timeout. Affects both v10.34.1 and v10.33.2 light cells, flaky.
Point the fleet (boot-1 + store) at quay.io/wakuorg/nwaku-pr:3939, the
build of logos-messaging/logos-delivery#3939 which prunes stale
Discv5-origin peers from the peer store (reconcilePeerStoreWithRoutingTable
in the discv5 search loop), so PX stops advertising dead sibling ENRs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(functional): fix fleet relay-mesh + filter rate limit for light-client compat
Two fleet-side root causes of tests-rpc-compat light-client flakiness, both
proven from CI fleet logs (compose.log), neither a status-go regression:
1. Relay-mesh fragmentation. nwaku runs AutoSharding, so each fleet node
relay-subscribes to ALL `num-shards-in-network` shards. At 257 the 2-node
gossipsub mesh is spread across 257 topics and rs/16/32 (the static community
shard) frequently has zero mesh peers, so a light client's lightpush returns
`not_published_to_any_peer` and the message never reaches the store. Evidence:
num=65 -> 0 lightpush failures; num=257 -> 101. Revert to 65 and drop the
--shard=128/256 subscriptions that forced it (status-go uses static 32/64;
the observed failures were all on rs/16/32, never 128/256).
2. Filter rate limit. The fleet set no --rate-limit, so nwaku enforced its stock
30/min/peer filter limit. Under `pytest -n 12` + reruns many light clients
subscribe concurrently and draw a 429, which go-waku turns into a permanent
subscription backoff for the whole run (dropped pushes / stuck community
joins). Lift filter+lightpush limits well above the test load.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(functional): point compat PEER_REFS at filter-rate-limit-fixed release branches
TEMP experiment on #7574: the remaining compat flakiness is filter 429s from
cross-test peer backends running released images that lack the filter full-node
rate-limit fix (default 1/s burst 1). Build the peer images from the release
branches WITH the fix cherry-picked in (tags compat-filterfix-10.34.x /
compat-filterfix-10.33.x) to see whether that removes the 429-driven drops.
Revert this override afterwards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(functional): run compat tests serially + fleet filter rate limit
Reduce the compat experiment to two fleet/test-harness changes (no status-go
code change):
- Run the compatibility marker serially (-n 0). They share one waku fleet, and
under pytest-xdist the fleet's peer-exchange hands each light client the ENRs
of other concurrent tests' torn-down backends, churning filter peer selection
and intermittently dropping messages (#7513).
- Lift the fleet nodes' stock filter/lightpush per-peer rate limits.
Reverts the earlier status-go go-waku filter limiter and the PEER_REFS override.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test(functional): bump compat test reruns to 6
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* persist Go module cache
fix: named volume approach
* like in nimbus-eth2
* Docker pre-creates /home/jenkins/gomodcache owned by jenkins
* named volume inherits owner on creation
* bind mount removed because chown in Dockerfile doesn't apply to bind mounts
* test(functional): add cross-version chat compatibility smoke tests
Run 1-1, group and community chat smoke tests with one participant on the
version under test and the other on a previous release backend built from a
git tag. Add a per-backend image override, a `compatibility` pytest marker,
a `test-functional-compatibility` make target, a Jenkins PR pipeline, and a
manual GitHub Actions workflow to validate on CI before merge.
* ci: trigger compatibility workflow on the feature branch
Add a push trigger for test/cross-version-chat-compat so the workflow runs on
CI before merge (workflow_dispatch only works once the file is on the default
branch). To be removed before merge.
* test(functional): address review on cross-version chat compat
- move peer-version resolution into the Makefile, selecting the latest
patch per minor (two newest minors)
- add TODO above the temporary worktree-based peer-image build
- wrap banUserFromCommunity in the python wakuext service
- exercise add/remove members from both vut and peer sides with 2-3
messages between each membership change; run the third node on the
peer image so every exchange crosses versions
* ci: default PEER_REFS for push runs of compatibility workflow
The GitHub Actions workflow invokes run_functional_tests.sh directly
(not via make), so the Makefile-resolved PEER_REFS is unavailable here
and push events have no workflow_dispatch input. Fall back to the same
default refs the manual dispatch already used.
* ci: keep compatibility tests within the time budget
The community smoke test runs two ~120s-budget community joins; under
the shared runner's reruns and two peer versions this pushed the job
past the 90-minute cap.
- make pytest reruns configurable (FUNCTIONAL_TESTS_RERUNS, default 2)
and use 1 for the compatibility marker
- raise the compatibility job timeout 90 -> 120 minutes (GitHub + Jenkins)
- run a single peer version in the GitHub pre-merge workflow; Jenkins
keeps the auto-resolved two latest minors
* ci: keep compatibility job timeout at 90 minutes
Reverts the 120-minute bump: with one peer version and reruns=1 the job
runs in ~21 min (tests ~2.5 min, no reruns observed across three runs),
so the original 90-minute cap has ample headroom.
* ci: drop unneeded Docker Hub registry auth from compat job
The dockerhub-statusteam-auto credential is not provisioned for PR jobs
and is not required: peer backends are built from git tags rather than
pulled, and the job pulls the same Docker Hub base images as tests-rpc,
which runs without Docker Hub auth. Mirror tests-rpc and keep only the
api-proxy-devel credential.
* fix(functional): use named logger in community_messages
The two start/after-send log calls used the root logger with %-style args.
The root logger carries SecretRedactingFilter, which preformats record.msg
but leaves record.args set, so the handler's `msg % args` raises TypeError.
Switch to the module logger (as the rest of the file does), which the root
filter does not intercept. Fixes community chat tests (compat and reliability).
* ci: remove temp github actions job
* ci: move PEER_REFS back to script
---------
Co-authored-by: Egor Rachkovskii <egorrachkovskii@status.im>
Co-authored-by: Igor Sirotin <sirotin@status.im>
* test: Refactor logging in `messenger.py` to use f-strings for improved readability
* test: Mark flaky light client tests in pytest and exclude them from reliability runs
* test: Add log dumping for docker compose in functional test scripts and enforce strict bash settings
* test: Enhance clone_and_run script with retries, strict bash settings, and export HTTP config
* test: avoid #7393 light-client filter race in accept_contact_request_and_wait
Mirror the existing 10s gate from join_community(): when sender is a Waku v2
light client, sleep before the receiver dispatches ACCEPT_CONTACT_REQUEST so
sender's filter subscription has time to propagate. Without it the ack is
delivered before the filter exists and is lost (#7393), causing
test_send_chat_messages_different_chats[wakuV2LightClient_True] and other
make_contacts()-based light-client tests to time out in tests-rpc.
---------
Co-authored-by: Egor Rachkovskii <egorrachkovskii@status.im>
* feat: support libstorage in the build system
* build: make sure linting includes logos-storage
* build: tweaking native build of libstorage that avoids calling make update (allowing to build on RYZEN processors)
* build: removes redundant scripts
* build: removes printing libstorage version on nix env
* test: more uniform naming (USE_LOGOS_STORAGE)
* build: bumps vendor hash
* test: include logosstorage tests in the coverage on the CI
* build: remove passing lib paths to the generate target
Co-authored-by: Jakub <jakub@status.im>
* build: Update Dockerfile - remove indentation
Co-authored-by: Jakub <jakub@status.im>
* build: remove redundant check in Makefile
Co-authored-by: Jakub <jakub@status.im>
* build: remove noise from Makefile
Co-authored-by: Jakub <jakub@status.im>
* build: remove redundant IFs from Makefile
* build: adds README to be used when printing logosstorage help messages
* build: move some Makefile vars to be better visible in context
---------
Co-authored-by: Jakub <jakub@status.im>
Igor Sirotin
