1139 Commits

Author SHA1 Message Date
Ivan FB
e58b5c15c8 chore: Better database query logs and logarithmic scale in grafana store panels (#3048) 2024-09-20 22:47:15 +03:00
Ivan FB
a16c64ad45 chore: extending store metrics (#3042)
* adding query_metrics module
* update fleet-dashboard with new store panels for better timing insight
2024-09-20 15:43:00 +03:00
Ivan FB
3921036ced sharding: reduce log level for a too spammy message (#3045) 2024-09-20 15:42:46 +03:00
NagyZoltanPeter
e7ae1a0382 chore: rate limit peer exchange protocol, enhanced response status in RPC (#3035)
* Enhanced peer-ex protocol - added rate limiting, added response status and desc to the rpc

* Better error result handling for PeerEx request, adjusted tests

* Refactored RateLimit configuration option for better CLI UX - now possible to set separate limits per protocol. Adjusted mountings. Added and adjusted tests

* Fix libwaku due to changes of error return type of fetchPeerExchangePeers

* Fix rate limit setting tests due to changed defaults

* Introduce new gauge to help dasboard effectively show current rate limit applied for protocol

* Adjust timeing in filter rate limit test to let macos CI test run ok.

* Address review findings, namings, error logs, removed left-overs

* Changes to reflect latest spec agreement and changes. PeerExchange RPC is changed the now respond structure will contain status_code and status_desc.
2024-09-18 15:58:07 +02:00
NagyZoltanPeter
6dfefc5e42 chore: Switch libnegentropy library build from shared to static linkage (#3041)
* Switch libnegentropy library build from shared to static linkage

* Update negentropy with -fPIC compile option that is necessary for libwaku build

* Bump waku-org/negentropy to the latest on master to incorporate merged static build of libnegentropy
2024-09-18 14:34:50 +02:00
gabrielmer
36df0fd838 fix: setting up node with modified config (#3036) 2024-09-16 16:30:38 +03:00
gabrielmer
2a596f4c77 adding a dynamic sleep interval in the connectivity loop (#3031) 2024-09-12 22:49:47 +02:00
Ivan FB
004b56e422 chore: libwaku - extending the library with peer_manager and peer_exchange features (#3026)
* libwaku: get peerids by protocol and peer exchange request
2024-09-11 10:13:54 +02:00
fryorcraken
723b009b20 chore: use submodule nph in CI to check lint (#3027) 2024-09-11 11:51:42 +10:00
gabrielmer
43bea3c476 chore: deprecating pubsub topic (#2997) 2024-09-10 15:07:12 -06:00
Ivan FB
f34a044ccf chore: lightpush - error metric less variable by only setting a fixed string (#3020) 2024-09-10 17:30:09 +02:00
Ivan FB
aefb7fb73d fix: get back health check for postgres legacy (#3010) 2024-09-10 15:07:53 +02:00
Ivan FB
5f2d87ec71 chore: Bump dependencies for v0.33 (#3017) 2024-09-09 10:45:14 +02:00
Ivan FB
50e15746d1 chore: enhance libpq management (#3015)
* db_postgres: register pg socket fd to chronos better data available awareness
* waku_store protocol: better logs to track time and new store metrics
2024-09-06 11:33:15 +02:00
Simon-Pierre Vivier
ed866321a0 chore: per limit split of PostgreSQL queries (#3008) 2024-09-04 10:17:28 -04:00
Ivan FB
236547ec7d chore: Better timing and requestId detail for slower store db queries (#2994)
* Better timing and requestId detail for store db queries slower than two seconds
* Adapt tests and client to allow sending custom store requestId
2024-08-29 22:56:14 +02:00
Ivan FB
97019896ac chore: remove unused setting from external_config.nim (#3004) 2024-08-29 17:54:37 +02:00
Ivan FB
6eff20507a libwaku: exposing more features (#3003)
- Allow to start or store discv5
- Expose lightpush request operation
- Expose list of connected and mesh peers
- Expose store client
2024-08-29 14:29:02 +02:00
Simon-Pierre Vivier
853025284b fix waku sync config defaults (#3001) 2024-08-28 10:26:38 -04:00
Ivan FB
f1db75262b chore: delivery monitor for store v3 reliability protocol (#2977)
- Use of observer observable pattern to inform delivery_monitor about subscription state
- send_monitor becomes a publish observer of lightpush and relay
- deliver monitor add more protection against possible crash and better logs
- creating a separate proc in store client for delivery monitor
2024-08-27 16:49:46 +02:00
Darshan K
fd653ef0da fix: peer-exchange issue (#2889) 2024-08-23 23:31:30 +05:30
Ivan FB
2e8f2f0076 chore: ANALYZE messages query should be performed regularly (#2986)
---------

Co-authored-by: Richard Ramos <info@richardramos.me>
2024-08-21 19:17:08 +02:00
NagyZoltanPeter
b68cc07261 Distinction between gross/net trafic in bandwidth per shard metric, added bandwidths and request rate panels to single node and fleet dashboards (#2920) 2024-08-21 17:10:29 +02:00
NagyZoltanPeter
3212459f77 chore: liteprotocoltester for simulation and for fleets (#2813)
* Added phase 2 - waku-simulatior integration in README.md

* Enhancement on statistics reports, added list of sent messages with hash, fixed latency calculations

* Enable standalonde running liteprotocoltester agains any waku network/fleet

* Fix missing env vars on run_tester_node.sh

* Adjustment on log levels, fix REST initialization

* Added standalon docker image build, fine tune duplicate detection and logging.

* Adjustments for waku-simulator runs

* Extended liteprotocoltester README.md with docker build

* Fix test inside docker service node connectivity failure

* Update apps/liteprotocoltester/README.md

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Explain minLatency calculation in code comment

---------

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-08-21 14:54:18 +02:00
gabrielmer
4b9bee99a8 chore: logging received message info via onValidated observer (#2973) 2024-08-19 14:13:28 +02:00
gabrielmer
3f641dff60 chore: deprecating protected topics in favor of protected shards (#2983) 2024-08-19 12:56:22 +02:00
gabrielmer
90b4dc89ff chore: rename NsPubsubTopic (#2974) 2024-08-19 11:29:35 +02:00
Ivan FB
73fbe5c337 postgres_driver limit max num hashes to 100 (#2976) 2024-08-19 11:12:31 +02:00
richΛrd
4a89875a36 chore: print WakuMessageHash as hex strings (#2969) 2024-08-14 21:04:20 +02:00
gabrielmer
1f3162ae5f avoid using the msg key in chronicles (#2970) 2024-08-14 16:40:08 +02:00
Simon-Pierre Vivier
301ce8068c feat: Nwaku Sync (#2403)
* feat: Waku Sync Protocol

* feat: state machine (#2656)

* feat: pruning storage mehcanism (#2673)

* feat: message transfer mechanism & tests (#2688)

* update docker files

* added ENR filed for sync & misc. fixes

* adding new sync range param & fixes

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com>
2024-08-13 07:27:34 -04:00
gabrielmer
459221f93d pruning excess in relay connections (#2965) 2024-08-12 17:59:11 +02:00
Ivan FB
6ae46c5fff faster retention policy when error and use of detach finalize when needed (#2966) 2024-08-12 10:47:01 +02:00
richΛrd
71946b911f fix: return on insert error (#2956) 2024-08-11 21:35:04 -04:00
Ivan FB
696587fdac chore: Optimize hash queries with lookup table (#2933)
* Upgrade Postgres schema to add messages_lookup table
* Perform optimized query for messageHash-only queries
2024-08-08 21:46:08 +02:00
Aaryamann Challani
b839b1c81f chore(keystore): verbose error message when credential is not found (#2943) 2024-08-07 11:57:03 +02:00
Darshan K
6748142f29 chore: upgrade peer exchange mounting (#2953) 2024-08-06 13:27:25 +05:30
Ivan FB
1ce87c49a8 lightpush better feedback in case the lightpush service node does not have peers (#2951) 2024-08-02 09:45:05 +02:00
gabrielmer
f10a604764 don't start node when it's discv5 only (#2947) 2024-08-01 23:28:00 +03:00
Prem Chaitanya Prathi
2614d93566 fix: network monitor improvements (#2939) 2024-07-30 16:56:49 +03:00
Ivan FB
f9552e133b chore: Simplification of store legacy code (#2931) 2024-07-30 14:05:23 +02:00
Simon-Pierre Vivier
2134ad76b4 feat: store resume (#2919)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-07-30 07:23:39 -04:00
Simon-Pierre Vivier
7fdabe5ad9 chore: add peer filtering by cluster for waku peer exchange (#2932) 2024-07-29 15:53:43 -04:00
Simon-Pierre Vivier
43e66939b5 fix: add back waku discv5 metrics (#2927)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-07-26 16:18:14 -04:00
Darshan K
48a79d3012 fix: update and shift unittest (#2934)
* fix: update and shift location of unit test
2024-07-26 16:57:34 +05:30
Sasha
1d850e43dc chore: return all connected peers from REST API (#2923)
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2024-07-23 12:58:56 +02:00
gabrielmer
aa9c30655c chore: adding lint job to the CI (#2925) 2024-07-23 13:57:24 +03:00
Darshan K
ad6f6c6bac fix: handle rln-relay-message-limit (#2867)
* fix: enforcing rln-contract max message limit and resolve early
2024-07-22 22:28:45 +05:30
gabrielmer
5d83d4a6fe setting filter handling logs to trace (#2914) 2024-07-17 13:29:53 +03:00
NagyZoltanPeter
ca634ef3ba feat: DOS protection of non relay protocols - rate limit phase3 (#2897)
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
  (although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
2024-07-16 15:46:21 +02:00