* Separate new lightpush protocol
New RPC defined
Rename al occurence of old lightpush to legacy lightpush, fix rest tests of lightpush
New lightpush protocol added back
Setup new lightpush protocol, mounting and rest api for it
modified: apps/chat2/chat2.nim
modified: tests/node/test_wakunode_lightpush.nim
modified: tests/node/test_wakunode_sharding.nim
modified: tests/test_peer_manager.nim
modified: tests/test_wakunode_lightpush.nim
renamed: tests/waku_lightpush/lightpush_utils.nim -> tests/waku_lightpush_legacy/lightpush_utils.nim
renamed: tests/waku_lightpush/test_all.nim -> tests/waku_lightpush_legacy/test_all.nim
renamed: tests/waku_lightpush/test_client.nim -> tests/waku_lightpush_legacy/test_client.nim
renamed: tests/waku_lightpush/test_ratelimit.nim -> tests/waku_lightpush_legacy/test_ratelimit.nim
modified: tests/wakunode_rest/test_all.nim
renamed: tests/wakunode_rest/test_rest_lightpush.nim -> tests/wakunode_rest/test_rest_lightpush_legacy.nim
modified: waku/factory/node_factory.nim
modified: waku/node/waku_node.nim
modified: waku/waku_api/rest/admin/handlers.nim
modified: waku/waku_api/rest/builder.nim
new file: waku/waku_api/rest/legacy_lightpush/client.nim
new file: waku/waku_api/rest/legacy_lightpush/handlers.nim
new file: waku/waku_api/rest/legacy_lightpush/types.nim
modified: waku/waku_api/rest/lightpush/client.nim
modified: waku/waku_api/rest/lightpush/handlers.nim
modified: waku/waku_api/rest/lightpush/types.nim
modified: waku/waku_core/codecs.nim
modified: waku/waku_lightpush.nim
modified: waku/waku_lightpush/callbacks.nim
modified: waku/waku_lightpush/client.nim
modified: waku/waku_lightpush/common.nim
modified: waku/waku_lightpush/protocol.nim
modified: waku/waku_lightpush/rpc.nim
modified: waku/waku_lightpush/rpc_codec.nim
modified: waku/waku_lightpush/self_req_handler.nim
new file: waku/waku_lightpush_legacy.nim
renamed: waku/waku_lightpush/README.md -> waku/waku_lightpush_legacy/README.md
new file: waku/waku_lightpush_legacy/callbacks.nim
new file: waku/waku_lightpush_legacy/client.nim
new file: waku/waku_lightpush_legacy/common.nim
new file: waku/waku_lightpush_legacy/protocol.nim
new file: waku/waku_lightpush_legacy/protocol_metrics.nim
new file: waku/waku_lightpush_legacy/rpc.nim
new file: waku/waku_lightpush_legacy/rpc_codec.nim
new file: waku/waku_lightpush_legacy/self_req_handler.nim
Adapt to non-invasive libp2p observers
cherry pick latest lightpush (v1) changes into legacy lightpush code after rebase to latest master
Fix vendor dependencies from origin/master after failed rebase of them
Adjust examples, test to new lightpush - keep using of legacy
Fixup error code mappings
Fix REST admin interface with distinct legacy and new lightpush
Fix lightpush v2 tests
* Utilize new publishEx interface of pubsub libp2p
* Adapt to latest libp2p pubslih design changes. publish returns an outcome as Result error.
* Fix review findings
* Fix tests, re-added lost one
* Fix rebase
* Apply suggestions from code review
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* Addressing review comments
* Fix incentivization tests
* Fix build failed on libwaku
* Change new lightpush endpoint version to 3 instead of 2. Noticed that old and new lightpush metrics can cause trouble in monitoring dashboards so decided to give new name as v3 for the new lightpush metrics and change legacy ones back - temporarly till old lightpush will be decommissioned
* Fixing flaky test with rate limit timing
* Fixing logscope of lightpush and legacy lightpush
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
* undo apt install libpcre (not circuit-relay related.)
* nat.nim: protect against possible exceptions when calling getExternalIP
* new external CLI argument, isRelayClient
* waku factory change to mount circuit hop proto by default
* waku_node: move autonat_service to a separate module
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
(although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
The id column is needed because it contains the message digest
which is used in store v2, and we need to keep support to
store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
We want the nwaku node to simultaneously support store-v2
requests and store-v3 requests.
Only the legacy archive is in charge of archiving messages, and the
archived information is suitable to fulfill both store-v2 and
store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed
---------
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
* chore: add unit test for testing duplicate message push with timedcache
* chore: update according to better naming convention
---------
Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
* chore: update logs with topic discription & debug msg
* chore: update unit according to error msg
* chore: update rest unit according to error msg
* chore: add content-topic with debug msg
* Separation of node health and initialization state from rln_relay status. Make (only) health endpoint avail early and install others in the last stage of node setup.
* Proper json report from /health, adjusted and fixed test, added convenient script for checking node health
* Stop wakunode2 if configured rest server cannot be started
* Fix wakuRlnRelay protocol existence check
* Fix typo
* Removed unused imports from touched files.
* Added missing /health test for all
* feat: Added simple, configurable rate limit for lightpush and store-query
Adjust lightpush rest response to rate limit, added tests ann some fixes
Add rest store query test for rate limit checks and proper error response
Update apps/wakunode2/external_config.nim
Move chronos/tokenbucket to nwaku codebasee with limited and fixed feature set
Add meterics counter to lightpush rate limits
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
* chore: remove deprecated legacy filter protocol
* fix: do not use legacy import in test
* fix: remove legacy test references
* fix: more test fixes, starting filter client
* fix: sigh. more references to remove.
* fix: fix dereferencing error
* fix: fix merge mess up
* fix: sigh. merge tool used tabs.
* fix: more peer manager tests needed fixing
---------
Co-authored-by: Hanno Cornelius <hanno@status.im>
Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
* chore(rln-relay-v2): additional testing
* fix: bump librln to v0.4.2 for v2
* fix: catch possible error from the copyFrom
* ci: rename step title for rln-version
* Add allowOrigin configuration for wakunode and WakuRestServer
Update nim-presto to the latest master that contains middleware support
Rework Rest Server in waku to utilize chronos' and presto's new middleware design and added proper CORS handling.
Added cors tests and fixes
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
A node that handles REST-Store requests normally acts as a
Store-client and therefore it retrieved the messages from another
Store-node.
With these changes, we allow a node with Store mounted, to retrieve
its messages. In other words, the node can act as a Store-server of
its messages.
* test_rest_store.nim: add a new test to validate that the self-node can
retrieve its messages to the REST client.
* rest/store/client.nim: add new proc to allow making a GET store
request without peerAddr.
* rest/store/handle.nim: add logic to handle requests that don't
provide peerAddr but the self/local node has Store mounted. In this case,
the self/local node will retrieve its locally stored messages.
* waku_store/self_req_handler.nim: logic to handle "store" requests
allowing the REST-store node to act as a Store-server node. The
'self_req_handler.nim' helps to bypass the store protocol and directly
retrieve the messages from the local/self node. I added this logic in
a separate file from 'protocol.nim' because it doesn't participate in
any libp2p communication.
* waku_store/protocol.nim: make 'queryHandler' attribute public so that
it can be used from the 'self_req_handler.nim' module.
* Refactor of FilterV2 subscription handling and maintenance with addition subscription time-to-live support.
Fixed all tests and reworked where subscription handling changes needed it.
Adapted REST API /admin filter subscription retrieve to new filter subscription structure.
* Fix tests and PR comments
* Added filter v2 subscription timeout tests and fixed
* Fix review comments and suggestions. No functional change.
* Remove leftover echoes from test_rest_admin
* Fix failed legacy filter tests due to separation of mounting the filters.
* Small fixes, fix naming typo, removed duplicated checks in test
This reverts commit dc1d6ce4bf7390e23b73d96634ff87ca9341e129.
We need to revert this commit because
the waku-simulator stopped working. i.e. the nodes couldn't establish
connections among them: 054ba9e33f
Also, the following js-waku test fails due to this commit:
"same cluster, different shard: nodes connect"
* waku_lightpush/protocol.nim: minor changes to make it compile after revert
* on_chain/group_manager: use .async: (raises:[Exception]).
* bump nim-dnsdisc
* update nim-chronos to the latest state
* chat2.nim: catch any possible exception when stopping
* chat2bridge.nim: make it to compile after vendor bump
* ValidIpAddress (deprecated) -> IpAddress
* vendor/nim-libp2p additional bump
* libwaku: adapt to vendor bump
* testlib/wakunode.nim: adapt to vendor bump (ValidIpAddress -> IpAddress)
* waku_node: avoid throwing any exception from stop*(node: WakuNode)
* test_confutils_envvar.nim: ValidIpAddress -> IpAddress
* test_jsonrpc_store: capture exception
* test_rln*: handling exceptions
* adaptation to make test_rln_* to work properly
* signature enhancement of group_manager methods
