Commit Graph

69 Commits

Author SHA1 Message Date
NagyZoltanPeter e2b7149f82
chore: Bump nimbus and nim to latest available - nim-2.0.12 (#3188)
* Bump nimbus and nim to latest available - nim-2.0.12
* Fix name collision of templates of result.nim and nwaku serdes.nim - unrecognizedFieldWarning
2024-12-10 14:42:54 +01:00
Simon-Pierre Vivier 2ab9c3d363
feat: remove Waku Sync 1.0 & Negentropy (#3185) 2024-11-29 09:09:41 -05:00
gabrielmer b3656d6eea
chore: naming connectPeer procedure (#3157) 2024-10-29 18:37:07 +02:00
Darshan K 729e63f53d
refactor: wrap peer store (#3051)
Encapsulate peerstore with wakupeerstore
2024-09-27 18:16:46 +05:30
fryorcraken ce9a8c468a
chore: use submodule nph in CI to check lint (#3027) 2024-09-11 11:51:42 +10:00
gabrielmer a3cd2a1a92
chore: deprecating pubsub topic (#2997) 2024-09-10 15:07:12 -06:00
Ivan FB 6c890de4a8
chore: Bump dependencies for v0.33 (#3017) 2024-09-09 10:45:14 +02:00
NagyZoltanPeter f4fa73e961
chore: liteprotocoltester for simulation and for fleets (#2813)
* Added phase 2 - waku-simulatior integration in README.md

* Enhancement on statistics reports, added list of sent messages with hash, fixed latency calculations

* Enable standalonde running liteprotocoltester agains any waku network/fleet

* Fix missing env vars on run_tester_node.sh

* Adjustment on log levels, fix REST initialization

* Added standalon docker image build, fine tune duplicate detection and logging.

* Adjustments for waku-simulator runs

* Extended liteprotocoltester README.md with docker build

* Fix test inside docker service node connectivity failure

* Update apps/liteprotocoltester/README.md

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Explain minLatency calculation in code comment

---------

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-08-21 14:54:18 +02:00
gabrielmer e8bce67d76
chore: logging received message info via onValidated observer (#2973) 2024-08-19 14:13:28 +02:00
gabrielmer 225b5e7260
avoid using the msg key in chronicles (#2970) 2024-08-14 16:40:08 +02:00
Simon-Pierre Vivier 2cc86c51da
feat: Nwaku Sync (#2403)
* feat: Waku Sync Protocol

* feat: state machine (#2656)

* feat: pruning storage mehcanism (#2673)

* feat: message transfer mechanism & tests (#2688)

* update docker files

* added ENR filed for sync & misc. fixes

* adding new sync range param & fixes

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com>
2024-08-13 07:27:34 -04:00
Sasha a29eca77a0
chore: return all connected peers from REST API (#2923)
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2024-07-23 12:58:56 +02:00
gabrielmer 086cc8edd2
chore: adding lint job to the CI (#2925) 2024-07-23 13:57:24 +03:00
NagyZoltanPeter ba418ab5ba
feat: DOS protection of non relay protocols - rate limit phase3 (#2897)
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
  (although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
2024-07-16 15:46:21 +02:00
Ivan FB fd6a71cdd7
chore: Bump dependencies for v0.31.0 (#2885)
* bump_dependencies.md: add nim-results dependency
* change imports stew/results to results
* switching to Nim 2.0.8
* waku.nimble: reflect the requirement nim 1.6.0 to 2.0.8
  Adding --mm:refc as nim 2.0 enables a new garbage collector that we're
  not yet ready to support
* adapt waku code to Nim 2.0
* gcsafe adaptations because Nim 2.0 is more strict
2024-07-09 13:14:28 +02:00
Darshan K 9bd8c33aee
chore: remove all pre-nim-1.6 deadcode from codebase (#2857) 2024-06-28 16:04:57 +05:30
gabrielmer 7205f95cff
chore: adding origin to peers admin endpoint (#2848) 2024-06-26 17:59:12 +02:00
Simon-Pierre Vivier a65b13fcb8
fix: invalid cursor returning messages (#2724)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-27 10:54:10 -04:00
Simon-Pierre Vivier 4a6ec468db
fix: store v3 bug fixes (#2718) 2024-05-23 08:01:52 -04:00
Ivan FB 176c98c589
standardize store types by using camel case instead of snake case (#2709) 2024-05-17 16:56:54 +02:00
Ivan FB 78132dc12e
chore: generic change to reduce the number of compilation warnings (#2696) 2024-05-16 22:29:11 +02:00
NagyZoltanPeter 026d804a0d
feat: Added flexible rate limit checks for store, legacy store and lightpush (#2668)
* Added flexible rate limit checks for store, legacy store and lightpush. Also added rate and traffic metrics.

* Fix chat2 after WakuLegacyStoreCodec rename

* Update waku/common/ratelimit.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Update waku/common/ratelimit.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Update waku/waku_store_legacy/protocol.nim

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Fix review findings, added limit to debug logs

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-05-09 20:07:49 +02:00
Ivan FB 404810aa8d
refactor: simplify app.nim and move discovery items to appropriate modules (#2657) 2024-05-01 21:13:08 +02:00
Simon-Pierre Vivier e03d1165e6
fix: store v3 validate cursor & remove messages (#2636) 2024-05-01 14:47:06 -04:00
Ivan FB 28389cf043
rest/store/types: contentTopic -> content_topic (#2646) 2024-04-29 16:19:07 +02:00
NagyZoltanPeter 2a4c0f1543
chore: Enabling to use a full node for lightpush via rest api without lightpush client configured (#2626)
* Enabling to use a full node for lightpush via rest api without light push client configured
2024-04-26 12:42:47 +02:00
Simon-Pierre Vivier 0b0fbfad5c
feat: store v3 (#2431) 2024-04-25 09:09:52 -04:00
Ivan FB b28207ab07
refactor: addition of waku_api/rest/builder.nim and reduce app.nim (#2623) 2024-04-24 15:59:50 +02:00
NagyZoltanPeter 6d135b0d1b
chore: Separation of node health and initialization state from rln_relay (#2612)
* Separation of node health and initialization state from rln_relay status. Make (only) health endpoint avail early and install others in the last stage of node setup.

* Proper json report from /health, adjusted and fixed test, added convenient script for checking node health

* Stop wakunode2 if configured rest server cannot be started

* Fix wakuRlnRelay protocol existence check

* Fix typo

* Removed unused imports from touched files.

* Added missing /health test for all
2024-04-23 18:53:18 +02:00
kaichao 768c61b114
fix: create options api for cors preflight request (#2598) 2024-04-18 18:29:50 +08:00
NagyZoltanPeter a00f350cd1
feat: Added simple, configurable rate limit for lightpush and store-query (#2390)
* feat: Added simple, configurable rate limit for lightpush and store-query
Adjust lightpush rest response to rate limit, added tests ann some fixes
Add rest store query test for rate limit checks and proper error response
Update apps/wakunode2/external_config.nim
Move chronos/tokenbucket to nwaku codebasee with limited and fixed feature set
Add meterics counter to lightpush rate limits

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
2024-04-15 15:28:35 +02:00
gabrielmer a04e0d9971
chore: bumping chronicles version (#2583) 2024-04-15 10:59:37 +03:00
Ivan FB d257855353
fix: rest store: content_topic -> contentTopic in the response (#2584) 2024-04-12 17:47:32 +02:00
Alvaro Revuelta e861317209
chore: remove deprecated legacy filter protocol (#2507)
* chore: remove deprecated legacy filter protocol

* fix: do not use legacy import in test

* fix: remove legacy test references

* fix: more test fixes, starting filter client

* fix: sigh. more references to remove.

* fix: fix dereferencing error

* fix: fix merge mess up

* fix: sigh. merge tool used tabs.

* fix: more peer manager tests needed fixing

---------

Co-authored-by: Hanno Cornelius <hanno@status.im>
Co-authored-by: Hanno Cornelius <68783915+jm-clius@users.noreply.github.com>
2024-03-25 18:07:56 +00:00
richΛrd 7aea2d4fc4
fix(rest): filter/v2/subscriptions response (#2529) 2024-03-18 18:21:06 -04:00
Ivan FB 3ba4378cbe
Generic re-style with nph 0.5.1 (#2396) 2024-03-16 00:08:47 +01:00
richΛrd c734f60d06
feat(rest): add support to ephemeral field (#2525) 2024-03-13 08:49:21 -04:00
Alvaro Revuelta 5a225809cd
fix: enable autosharding in any cluster (#2505) 2024-03-13 10:58:13 +01:00
Aaryamann Challani 059cb97518
chore(rln-relay-v2): wakunode testing + improvements (#2501)
* chore(rln-relay-v2): additional testing

* fix: bump librln to v0.4.2 for v2

* fix: catch possible error from the copyFrom

* ci: rename step title for rln-version
2024-03-12 16:20:30 +05:30
Ivan FB b890f9c663
rest: rm openapi defs. they are in https://github.com/waku-org/waku-rest-api (#2520) 2024-03-08 19:43:55 +01:00
NagyZoltanPeter d832f92a43
chore: Implemented CORS handling for nwaku REST server (#2470)
* Add allowOrigin configuration for wakunode and WakuRestServer
Update nim-presto to the latest master that contains middleware support
Rework Rest Server in waku to utilize chronos' and presto's new middleware design and added proper CORS handling.
Added cors tests and fixes

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-02-29 09:48:14 +01:00
gabrielmer 3903f130cb
feat: supporting meta field in WakuMessage (#2384) 2024-02-14 17:29:59 +02:00
Aaryamann Challani 50308eda0d
feat(rln-relay-v2): nonce/messageId manager (#2413)
* feat(rln-relay-v2): nonce/messageId manager

* fix: simplify
2024-02-13 10:18:02 +05:30
gabrielmer e4e147bcbb
feat: running validators in /relay/v1/auto/messages/{topic} (#2394) 2024-02-05 10:24:54 +02:00
gabrielmer 59d8b6204f
feat: running validators in /relay/v1/messages/{pubsubTopic} (#2373) 2024-02-01 18:16:10 +01:00
Ivan FB 3e65cc18f6
REST store: get msgs from self node when store is mounted and no peerAddr is passed (#2387)
A node that handles REST-Store requests normally acts as a 
Store-client and therefore it retrieved the messages from another
Store-node.
With these changes, we allow a node with Store mounted, to retrieve
its messages. In other words, the node can act as a Store-server of
its messages.

* test_rest_store.nim: add a new test to validate that the self-node can
retrieve its messages to the REST client.

* rest/store/client.nim: add new proc to allow making a GET store
request without peerAddr.

* rest/store/handle.nim: add logic to handle requests that don't
provide peerAddr but the self/local node has Store mounted. In this case,
the self/local node will retrieve its locally stored messages.

* waku_store/self_req_handler.nim: logic to handle "store" requests
allowing the REST-store node to act as a Store-server node. The
'self_req_handler.nim' helps to bypass the store protocol and directly
retrieve the messages from the local/self node. I added this logic in
a separate file from 'protocol.nim' because it doesn't participate in
any libp2p communication.

* waku_store/protocol.nim: make 'queryHandler' attribute public so that
it can be used from the 'self_req_handler.nim' module.
2024-01-31 17:43:59 +01:00
Simon-Pierre Vivier edca1df1a8
feat: shard aware relay peer management (#2332)
note that this feature is behind a config flag. `--relay-shard-manager`
2024-01-30 07:28:21 -05:00
gabrielmer f841454edc
chore: improve POST /relay/v1/auto/messages/{topic} error handling (#2339) 2024-01-18 13:49:13 +01:00
NagyZoltanPeter c3358409bb
chore: Refactor of FilterV2 subscription management with Time-to-live maintenance (#2341)
* Refactor of FilterV2 subscription handling and maintenance with addition subscription time-to-live support.
Fixed all tests and reworked where subscription handling changes needed it.
Adapted REST API /admin filter subscription retrieve to new filter subscription structure.

* Fix tests and PR comments

* Added filter v2 subscription timeout tests and fixed

* Fix review comments and suggestions. No functional change.

* Remove leftover echoes from test_rest_admin

* Fix failed legacy filter tests due to separation of mounting the filters.

* Small fixes, fix naming typo, removed duplicated checks in test
2024-01-16 17:27:40 +01:00
gabrielmer bcab9d55da
feat: adding filter data admin endpoint (REST) (#2314) 2024-01-09 11:42:29 +01:00