Commit Graph

1135 Commits

Author SHA1 Message Date
Ivan FB 5ea1cf0cf3
chore: libwaku - extending the library with peer_manager and peer_exchange features (#3026)
* libwaku: get peerids by protocol and peer exchange request
2024-09-11 10:13:54 +02:00
fryorcraken ce9a8c468a
chore: use submodule nph in CI to check lint (#3027) 2024-09-11 11:51:42 +10:00
gabrielmer a3cd2a1a92
chore: deprecating pubsub topic (#2997) 2024-09-10 15:07:12 -06:00
Ivan FB d3e6717a60
chore: lightpush - error metric less variable by only setting a fixed string (#3020) 2024-09-10 17:30:09 +02:00
Ivan FB 5a0edff79a
fix: get back health check for postgres legacy (#3010) 2024-09-10 15:07:53 +02:00
Ivan FB 6c890de4a8
chore: Bump dependencies for v0.33 (#3017) 2024-09-09 10:45:14 +02:00
Ivan FB 45319f09c9
chore: enhance libpq management (#3015)
* db_postgres: register pg socket fd to chronos better data available awareness
* waku_store protocol: better logs to track time and new store metrics
2024-09-06 11:33:15 +02:00
Simon-Pierre Vivier e1e05afb02
chore: per limit split of PostgreSQL queries (#3008) 2024-09-04 10:17:28 -04:00
Ivan FB e8a49b76b2
chore: Better timing and requestId detail for slower store db queries (#2994)
* Better timing and requestId detail for store db queries slower than two seconds
* Adapt tests and client to allow sending custom store requestId
2024-08-29 22:56:14 +02:00
Ivan FB fd84363ec4
chore: remove unused setting from external_config.nim (#3004) 2024-08-29 17:54:37 +02:00
Ivan FB bd2cf99d7e
libwaku: exposing more features (#3003)
- Allow to start or store discv5
- Expose lightpush request operation
- Expose list of connected and mesh peers
- Expose store client
2024-08-29 14:29:02 +02:00
Simon-Pierre Vivier 87b340f78a
fix waku sync config defaults (#3001) 2024-08-28 10:26:38 -04:00
Ivan FB 0f68274c85
chore: delivery monitor for store v3 reliability protocol (#2977)
- Use of observer observable pattern to inform delivery_monitor about subscription state
- send_monitor becomes a publish observer of lightpush and relay
- deliver monitor add more protection against possible crash and better logs
- creating a separate proc in store client for delivery monitor
2024-08-27 16:49:46 +02:00
Darshan K 4315710289
fix: peer-exchange issue (#2889) 2024-08-23 23:31:30 +05:30
Ivan FB 32f2d85dcc
chore: ANALYZE messages query should be performed regularly (#2986)
---------

Co-authored-by: Richard Ramos <info@richardramos.me>
2024-08-21 19:17:08 +02:00
NagyZoltanPeter c7093d7ce2
Distinction between gross/net trafic in bandwidth per shard metric, added bandwidths and request rate panels to single node and fleet dashboards (#2920) 2024-08-21 17:10:29 +02:00
NagyZoltanPeter f4fa73e961
chore: liteprotocoltester for simulation and for fleets (#2813)
* Added phase 2 - waku-simulatior integration in README.md

* Enhancement on statistics reports, added list of sent messages with hash, fixed latency calculations

* Enable standalonde running liteprotocoltester agains any waku network/fleet

* Fix missing env vars on run_tester_node.sh

* Adjustment on log levels, fix REST initialization

* Added standalon docker image build, fine tune duplicate detection and logging.

* Adjustments for waku-simulator runs

* Extended liteprotocoltester README.md with docker build

* Fix test inside docker service node connectivity failure

* Update apps/liteprotocoltester/README.md

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>

* Explain minLatency calculation in code comment

---------

Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-08-21 14:54:18 +02:00
gabrielmer e8bce67d76
chore: logging received message info via onValidated observer (#2973) 2024-08-19 14:13:28 +02:00
gabrielmer e51ffe0759
chore: deprecating protected topics in favor of protected shards (#2983) 2024-08-19 12:56:22 +02:00
gabrielmer 67439057fb
chore: rename NsPubsubTopic (#2974) 2024-08-19 11:29:35 +02:00
Ivan FB b8550c5550
postgres_driver limit max num hashes to 100 (#2976) 2024-08-19 11:12:31 +02:00
richΛrd 2fd4eb62fd
chore: print WakuMessageHash as hex strings (#2969) 2024-08-14 21:04:20 +02:00
gabrielmer 225b5e7260
avoid using the msg key in chronicles (#2970) 2024-08-14 16:40:08 +02:00
Simon-Pierre Vivier 2cc86c51da
feat: Nwaku Sync (#2403)
* feat: Waku Sync Protocol

* feat: state machine (#2656)

* feat: pruning storage mehcanism (#2673)

* feat: message transfer mechanism & tests (#2688)

* update docker files

* added ENR filed for sync & misc. fixes

* adding new sync range param & fixes

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com>
2024-08-13 07:27:34 -04:00
gabrielmer 54b5222222
pruning excess in relay connections (#2965) 2024-08-12 17:59:11 +02:00
Ivan FB 71d4a11647
faster retention policy when error and use of detach finalize when needed (#2966) 2024-08-12 10:47:01 +02:00
richΛrd 5f0fbd78c4
fix: return on insert error (#2956) 2024-08-11 21:35:04 -04:00
Ivan FB 4d47aa655f
chore: Optimize hash queries with lookup table (#2933)
* Upgrade Postgres schema to add messages_lookup table
* Perform optimized query for messageHash-only queries
2024-08-08 21:46:08 +02:00
Aaryamann Challani 0f11ee1422
chore(keystore): verbose error message when credential is not found (#2943) 2024-08-07 11:57:03 +02:00
Darshan K 42f1bed04c
chore: upgrade peer exchange mounting (#2953) 2024-08-06 13:27:25 +05:30
Ivan FB ebda56dd90
lightpush better feedback in case the lightpush service node does not have peers (#2951) 2024-08-02 09:45:05 +02:00
gabrielmer 64855502cf
don't start node when it's discv5 only (#2947) 2024-08-01 23:28:00 +03:00
Prem Chaitanya Prathi 8058323760
fix: network monitor improvements (#2939) 2024-07-30 16:56:49 +03:00
Ivan FB d4e8a0dab6
chore: Simplification of store legacy code (#2931) 2024-07-30 14:05:23 +02:00
Simon-Pierre Vivier aed2a1130e
feat: store resume (#2919)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-07-30 07:23:39 -04:00
Simon-Pierre Vivier b4618f98ed
chore: add peer filtering by cluster for waku peer exchange (#2932) 2024-07-29 15:53:43 -04:00
Simon-Pierre Vivier e4e01fabfe
fix: add back waku discv5 metrics (#2927)
Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
2024-07-26 16:18:14 -04:00
Darshan K 08973add7e
fix: update and shift unittest (#2934)
* fix: update and shift location of unit test
2024-07-26 16:57:34 +05:30
Sasha a29eca77a0
chore: return all connected peers from REST API (#2923)
* Remove the condition of gathering connected peers with relay and user req/resp protocols.
* Return PeerExchange protocol support of connected nodes with /admin/peers
* Added test for checking return of PeerExchange mounted protocol of connected peer by GET /admin/peers

---------

Co-authored-by: NagyZoltanPeter <113987313+NagyZoltanPeter@users.noreply.github.com>
2024-07-23 12:58:56 +02:00
gabrielmer 086cc8edd2
chore: adding lint job to the CI (#2925) 2024-07-23 13:57:24 +03:00
Darshan K 8d107b0ded
fix: handle rln-relay-message-limit (#2867)
* fix: enforcing rln-contract max message limit and resolve early
2024-07-22 22:28:45 +05:30
gabrielmer 5c539fe13d
setting filter handling logs to trace (#2914) 2024-07-17 13:29:53 +03:00
NagyZoltanPeter ba418ab5ba
feat: DOS protection of non relay protocols - rate limit phase3 (#2897)
* DOS protection of non relay protocols - rate limit phase3:
- Enhanced TokenBucket to be able to add compensation tokens based on previous usage percentage,
- per peer rate limiter 'PeerRateLimier' applied on waku_filter_v2 with opinionated default of acceptable request rate
- Add traffic metrics to filter message push
- RequestRateLimiter added to combine simple token bucket limiting of request numbers but consider per peer usage over time and prevent some peers to over use the service
  (although currently rule violating peers will not be disconnected by this time only their requests will get not served)
- TimedMap utility created (inspired and taken from libp2p TimedCache) which serves as forgiving feature for peers had been overusing the service.
- Added more tests
- Fix rebase issues
- Applied new RequestRateLimiter for store and legacy_store and lightpush
* Incorporate review comments, typos, file/class naming and placement changes.
* Add issue link reference of the original issue with nim-chronos TokenBucket
* Make TimedEntry of TimedMap private and not mixable with similar named in libp2p
* Fix review comments, renamings, const instead of values and more comments.
2024-07-16 15:46:21 +02:00
Ivan FB 71ee42de56
chore: simple PR to enhance postgres and retention policy logs (#2884) 2024-07-15 20:58:31 +02:00
Ivan FB 241fb8cb97
chore: Update master from release v0.30 (#2908)
* chore(rln): rln message limit to 100 (#2883)
* postgres_driver: add more error handling when creating partitions
   Given that multiple nodes can be connected to the same database,
   it can happen that other node did something that my node was willing
   to do. In this commit, we overcome the possible "interleaved" 
   partition creation.

---------

Co-authored-by: Alvaro Revuelta <alvrevuelta@gmail.com>
2024-07-15 18:00:44 +02:00
Ivan FB b71ab28517
archive legacy run report metrics every 30 minutes instead of 1 (#2903) 2024-07-15 17:39:21 +02:00
gabrielmer db8ac5a83e
logging content topic of spam messages (#2906) 2024-07-15 16:43:53 +03:00
gabrielmer 8578fb0c3e
chore: improving logging under debugDiscv5 flag (#2899) 2024-07-15 10:55:31 +03:00
Simon-Pierre Vivier f54ba10bc7
chore(archive): archive and drivers refactor (#2761)
* queue driver refactor (#2753)
* chore(archive): archive refactor (#2752)
* chore(archive): sqlite driver refactor (#2754)
* chore(archive): postgres driver refactor (#2755)
* chore(archive): renaming & copies (#2751)
* posgres legacy: stop using the storedAt field
* migration script 6: we still need the id column
  The id column is needed because it contains the message digest
  which is used in store v2, and we need to keep support to
  store v2 for a while
* legacy archive: set target migration version to 6
* waku_node: try to use wakuLegacyArchive if wakuArchive is nil
* node_factory, waku_node: mount legacy and future store simultaneously
  We want the nwaku node to simultaneously support store-v2
  requests and store-v3 requests.
  Only the legacy archive is in charge of archiving messages, and the
  archived information is suitable to fulfill both store-v2 and
  store-v3 needs.
* postgres_driver: adding temporary code until store-v2 is removed

---------

Co-authored-by: Ivan FB <128452529+Ivansete-status@users.noreply.github.com>
Co-authored-by: gabrielmer <101006718+gabrielmer@users.noreply.github.com>
Co-authored-by: Ivan Folgueira Bande <ivansete@status.im>
2024-07-12 18:19:12 +02:00
Aaryamann Challani 1c9eb27415
fix(rln_keystore_generator): improve error handling for unrecoverable failure (#2881) 2024-07-10 19:12:49 +02:00