* postgres: first step to implement partition management
* postgres_driver: use of times.now().toTime().toUnix() instead of Moment.now()
* postgres migrations: set new version to 2
* test_driver_postgres: use of assert instead of require and avoid using times.now()
* postgres_driver: better implementation of the reset method with partitions
* Remove createMessageTable, init, and deleteMessageTable procs
* postgres: ensure we use the version 15.4 in tests
* postgres_driver.nim: enhance debug logs partition addition
* ci.yml: ensure logs are printed without colors
* postgres_driver: starting the loop factory in an asynchronous task
* postgres_driver: log partition name and size when removing a partition
* Add postgres_driver/migrations.nim
* Postgres and archive logic adaptation to the migration implementation
* libwaku: adapt node_lifecycle_request.nim to migration refactoring
* test_app.nim: add more detail for test that only fails in CI
* postgres migrations: store the migration scripts inside the resulting wakunode binary instead of external .sql files.
* Add allowOrigin configuration for wakunode and WakuRestServer
Update nim-presto to the latest master that contains middleware support
Rework Rest Server in waku to utilize chronos' and presto's new middleware design and added proper CORS handling.
Added cors tests and fixes
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* new proc to foster different size retention policy implementations
The new proc, decreaseDatabaseSize, will have different implementations
per each driver. For example, in future commits we will implement a size
retention policy thanks to partitions management, in Postgres.
* RetentionPolicy: use of new instead of init for ref object types
* waku_archive: fix signatures in decreaseDatabaseSize methods
* retention_policy_size: minor cleanup of comments and imports
* chore(rln-relay-v2): add tests for serde
* fix: call isOk fn instead of prop access
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* fix: make cast more explicit
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Add ligthpush payload tests.
* Add end to end lightpush tests.
* updating vendor/nim-unittest2 to protect against core dump issue
* Enable "Valid Payload Sizes" test again
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
A node that handles REST-Store requests normally acts as a
Store-client and therefore it retrieved the messages from another
Store-node.
With these changes, we allow a node with Store mounted, to retrieve
its messages. In other words, the node can act as a Store-server of
its messages.
* test_rest_store.nim: add a new test to validate that the self-node can
retrieve its messages to the REST client.
* rest/store/client.nim: add new proc to allow making a GET store
request without peerAddr.
* rest/store/handle.nim: add logic to handle requests that don't
provide peerAddr but the self/local node has Store mounted. In this case,
the self/local node will retrieve its locally stored messages.
* waku_store/self_req_handler.nim: logic to handle "store" requests
allowing the REST-store node to act as a Store-server node. The
'self_req_handler.nim' helps to bypass the store protocol and directly
retrieve the messages from the local/self node. I added this logic in
a separate file from 'protocol.nim' because it doesn't participate in
any libp2p communication.
* waku_store/protocol.nim: make 'queryHandler' attribute public so that
it can be used from the 'self_req_handler.nim' module.
* Refactor of FilterV2 subscription handling and maintenance with addition subscription time-to-live support.
Fixed all tests and reworked where subscription handling changes needed it.
Adapted REST API /admin filter subscription retrieve to new filter subscription structure.
* Fix tests and PR comments
* Added filter v2 subscription timeout tests and fixed
* Fix review comments and suggestions. No functional change.
* Remove leftover echoes from test_rest_admin
* Fix failed legacy filter tests due to separation of mounting the filters.
* Small fixes, fix naming typo, removed duplicated checks in test
* message.nim: set max message size to 150KiB according to spec
Using KiB instead of KB because that seems more aligned with
the actual default defined in nim-libp2p (1024 * 1024)
Spec details: https://rfc.vac.dev/spec/64/#message-size
* test_protocol.nim: align test to current WakuMessage limit
* test_waku_client.nim: adapt test to MaxWakuMessageSize change
* make maxMessageSize configurable for wakunode2
* wakunode2 app now accepts max-num-bytes-msg-size with KiB, KB, or B units
* testlib/wakunode.nim: set maxMessageSize: "1024 KiB"
* test_waku_client.nim: remove duplicate check in "Valid Payload Sizes"
* set DefaultMaxWakuMessageSizeStr as the only source of truth
* external_config.nim: rename max-num-bytes-msg-size -> max-msg-size
This reverts commit dba9820c1f.
We need to revert this commit because
the waku-simulator stopped working. i.e. the nodes couldn't establish
connections among them: 054ba9e33f
Also, the following js-waku test fails due to this commit:
"same cluster, different shard: nodes connect"
* waku_lightpush/protocol.nim: minor changes to make it compile after revert
* on_chain/group_manager: use .async: (raises:[Exception]).
* bump nim-dnsdisc
* update nim-chronos to the latest state
* chat2.nim: catch any possible exception when stopping
* chat2bridge.nim: make it to compile after vendor bump
* ValidIpAddress (deprecated) -> IpAddress
* vendor/nim-libp2p additional bump
* libwaku: adapt to vendor bump
* testlib/wakunode.nim: adapt to vendor bump (ValidIpAddress -> IpAddress)
* waku_node: avoid throwing any exception from stop*(node: WakuNode)
* test_confutils_envvar.nim: ValidIpAddress -> IpAddress
* test_jsonrpc_store: capture exception
* test_rln*: handling exceptions
* adaptation to make test_rln_* to work properly
* signature enhancement of group_manager methods
* db_postgres: use prepared statements on most freq select queries
* db_postgres/dbconn.nim adding better feedback in case of query error
* dbconn: use of isOkOr
* pgasyncpool: refactor to reduce code (valueOr, catch:)
/admin rest api implementation and tests
* open api doc
* Add rest admin test to all tests
* Enrich /admin get peers interface, group protocols by peers in response
* This reverts former change on lighpush error handling, now zero peer publish still succeed. This will allow js-waku use case to succeed.
* Adjust lightpush rest-api test
* chore: add retention policy with GB or MB limitation #1885
* chore: add retention policy with GB or MB limitation
* chore: updated code post review- retention policy
* ci: extract discordNotify to separate file
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* ci: push images to new wakuorg/nwaku repo
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* ci: enforce default Docker image tags strictly
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* ci: push GIT_REF if it looks like a version
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* fix: update wakuv2 fleet DNS discovery enrtree
https://github.com/status-im/infra-misc/issues/171
* chore: resolving DNS IP and publishing it when no extIp is provided (#2030)
* feat(coverage): Add simple coverage (#2067)
* Add test aggregator to all directories.
* Implement coverage script.
* fix(ci): fix name of discord notify method
Also use absolute path to load Groovy script.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version (#2080)
* chore(networkmonitor): refactor setConnectedPeersMetrics, make it partially concurrent, add version
* add more metrics, refactor how most metrics are calculated
* rework metrics table fillup
* reset connErr to make sure we honour successful reconnection
* chore(cbindings): Adding cpp example that integrates the 'libwaku' (#2079)
* Adding cpp example that integrates the `libwaku`
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* fix(ci): update the dependency list in pre-release WF (#2088)
* chore: adding NetConfig test suite (#2091)
---------
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Anton Iakimov <yakimant@gmail.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Álex Cabeza Romero <alex93cabeza@gmail.com>
Co-authored-by: Vaclav Pavlin <vaclav@status.im>
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
RestApi Lightpush endpoint implemented
* Openapi definition for lightpush rest api
* Update waku/node/rest/lightpush/handlers.nim
* Fix install handler naming, added negative test cases and fixes for restapi lightpush
* Fix error handling in lightpush rest handler
* Fix main success case - relay message that comes with pushRequest
* Fix rest relay serdes test with RelayWakuMessage validation changes
* Checking response message tests
Filter v2 rest api support implemented
Filter rest api documentation updated with v1 and v2 interface support.
Separated legacy filter rest interface
Fix code and tests of v2 Filter rest api
Filter v2 message push test added
Applied autoshard to Filter V2
Redesigned FilterPushHandling, code style, catch up apps and tests with filter v2 interface changes
Rename of FilterV1SubscriptionsRequest to FilterLegacySubscribeRequest, fix broken chat2 app, fix tests
Changed Filter v2 push handler subscription to simple register
Separate node's filterUnsubscribe and filterUnsubscribeAll
* Move waku filter utility functions to a different file for reuse.
* Add asyncSetup and asyncTeardown templates.
* Implement waku filter subscriber ping tests.
* removing implicit dependency with libpq if postgres is not being used.
* We only run the postgres tests when explicitly willing to, i.e. make
POSTGRES=1 test. The reason is that the postgres tests expect a
database instance to be running locally.
* chore: remove v1 code
* chore: deprecate support for v1 compatibility from JSON-RPC API
* chore: remove v1 tests from JSON-RPC suite
* chore: remove wakubridge code
* chore: remove v1 compatibility functions
* chore: remove support for v1 payloads from chat2 app
* chore: remove v1 from CI processes
* fix: lingering references to private API functions
* fix: remove v1 compat from chat2
* fix: some more lingering references in tests
* chore(rln-relay): Verify proofs based on bandwidth usage
* fix(rln-relay): make default threshold 0, for backwards compat
* fix(rln-relay): add unit test for bandwidth cutoff
* fix(rln-relay): rlnRelayBandwidthTreshold option
- Remove discv5 from node and it's builder
- Require a record to init a node (was previously using discv5 record).
- Fix various tests that assumed nodes with discv5.
- Fix ENR creation
- Fix wakubridge
- Fix networkmonitor
* Fixing issue that prevented the wakunode2 from starting
The issue was introduced in PR#1818.
Before this commit, the `wakunode2` app crashed with the next error:
ERR 2023-06-27 15:57:27.268+00:00 5/7 Starting node and protocols failed
topics="wakunode main" tid=1 file=wakunode2.nim:92 error="failed to
start waku discovery v5: "
* fix tests accordingly
* publisher.nim, subscriber.nim: fix
* Refactoring the Waku Archive. Simplifying the app.nim
This change is needed to accommodate the further PRs where we will integrate Postgres in `wakunode2`.
* feat(discv5): added find random nodes with predicate (#1762)
Co-authored-by: Lorenzo Delgado <lnsdev@proton.me>
* chore: play around with sleep times on tests
* chore: simplify flaky tests
* chore: update waku/v2/node/waku_node.nim
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
* chore: update tests/v2/test_waku_discv5.nim
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
* chore: update examples/v2/subscriber.nim
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
---------
Co-authored-by: Lorenzo Delgado <lnsdev@proton.me>
Co-authored-by: Ivan Folgueira Bande <128452529+Ivansete-status@users.noreply.github.com>
* Add postgres basic support
* test_driver_postgres.nim: adding simple test to validate asynchronous-ness
* Aligning the changes made by Andrea to the current master state
* test_driver_postgres.nim: new unit test (checking duplicate insertion)
---------
Co-authored-by: Andrea Maria Piana <andrea.maria.piana@gmail.com>
gabrielmer
Aaryamann Challani
Ivan FB
NagyZoltanPeter
Simon-Pierre Vivier
Alvaro Revuelta
Álex Cabeza Romero
Hanno Cornelius
Abhimanyu
Sergei Tikhomirov
Vaclav Pavlin