Commit Graph

414 Commits

Author SHA1 Message Date
Prem Chaitanya Prathi 18c16de94e
fix: modify store test to not depend on order of msgs (#832)
* fix: modify store test to not depend on order of msgs

Co-authored-by: richΛrd <info@richardramos.me>

---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-10-25 03:41:00 +05:30
richΛrd 94f18c537c refactor: validate protobuffers for lightpush and relay (#824) 2023-10-24 17:48:22 -04:00
Richard Ramos fa51d10b4b fix: stream close/reset (#823) 2023-10-24 17:48:22 -04:00
Prem Chaitanya Prathi a3c3aab44e
fix: propagate relay subscribe opt properly (#829) 2023-10-25 02:11:42 +05:30
richΛrd 077df2fbb6
fix: do not disconnect peers in `TestWakuMetadataRequest` (#826) 2023-10-24 16:01:11 -04:00
Richard Ramos 9161c4f7fe fix: stream closing/reset 2023-10-20 20:30:23 -04:00
Richard Ramos 519fa2977a refactor: rename `connOpt` to `stream` 2023-10-20 20:30:23 -04:00
Richard Ramos 19ba25ffcb feat: metadata protocol 2023-10-20 20:30:23 -04:00
richΛrd 4181655b7a
fix: msg digest matches msg hash (#820) 2023-10-20 16:25:54 -04:00
Prem Chaitanya Prathi b5be83a02e
feat : autoshard relay api (#807)
* fix: using relay without bcaster should consume and drop messages

* update relay api usage

* move subscription to broadcaster

* move filter logic under subscription

* Support more than 1 relay subscription for a pubSubTopic

* modify relay Publish API to derive pubSubTopic based on autosharding

* implement relay RPC methods for autosharding

* remove relay msgChannel and relay on pubsub buffersize for subscription

Co-authored-by: richΛrd <info@richardramos.me>

* handle relay subscribe with noConsumer and address issue reported in code review

* chore: reorg relay code

---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-10-21 01:26:18 +05:30
Prem Chaitanya Prathi ac9d826b03
fix: handle empty content topics in filter subcribe and unsubscribe (#812)
* fix: handle empty content topics in filter subcribe and unsubscribe
2023-10-18 01:23:40 +05:30
Richard Ramos d4abe15634 fix: addr update 2023-10-17 09:00:30 -04:00
Richard Ramos ee94581d0a feat: prefer circuit relay addr over ws, and update cache with enrs with newer seq number 2023-10-17 08:29:24 -04:00
Prem Chaitanya Prathi 6955d01498
Update peer selection options for light protocols (#787)
* Update peer selection options for lightPush

* Update peer selection options for filter

* migrate peer selection functionality from peer manager

Co-authored-by: richΛrd <info@richardramos.me>

---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-10-16 22:12:01 +05:30
Richard Ramos 7826e31f14 fix(store): query time comparison and max rows per page 2023-10-12 10:58:55 -04:00
richΛrd 3aa477cbc6
fix: return errors in FilterSubscribeOption (#794) 2023-10-07 16:19:53 -04:00
richΛrd dcc828749f
fix: use https for cloning submodules instead of ssh (#780)
* fix: use https for cloning submodules instead of ssh
* chore: move to libs/

This simplifies the checkout process for github actions
2023-09-30 13:57:49 -04:00
Prem Chaitanya Prathi 47c961dcbb
feat: update lightpush API for autosharding (#774)
* feat: update lightpush API to make pubSubTopic optional as per autosharding

* Extract contentFilter and subscriptions out of filter to reuse in relay (#779)

* chore: extract contentFilter outside filter package

* chore: move subscription outside of filter so that it can be modified and reused for relay

* Feat: filter select peer for sharding (#783)

* update selectPeer to support pubsubTopic based selection
2023-09-29 10:43:25 +05:30
Prem Chaitanya Prathi 388f56b43f
feat: Sharded peer management - Relay (#764)
* feat: connect/disconnect to peers based on node topic sub/unsub

* feat: maintain healty relay connections per pubSubTopic

Co-authored-by: richΛrd <info@richardramos.me>

* chore: add config to limit peerstore capacity (#770)
2023-09-27 12:16:37 +05:30
richΛrd d324234c81
fix(filter2): add requestID to pings and remove unneeded log (#776) 2023-09-26 12:27:29 -04:00
Vitaliy Vlasov 16ec22596e feat: change UnsubscribeWithSubscription so that it's single sub-specific
Also merge FilterSubscribe and FilterUnsubscribe options/params
2023-09-22 17:53:33 +03:00
Prem Chaitanya Prathi 054bdae1de
Feat/autoshard filter (#723)
* feat: update filter client to support autosharding

* chore: add filter tests for autoshard

* chore:update filter API docs for autosharding

* chore: docs changes to indicate sharding impact on pubSubTopic

* fix: handle partial errors during subscribe and return failed content-topic details
2023-09-20 11:26:55 +05:30
Vitaliy Vlasov 81638fe111
Use PubsubTopic naming; enforce unique ContentTopics (#750) 2023-09-19 18:22:11 +05:30
Prem Chaitanya Prathi 9b05d48318
Feat : handle dynamic peer topic sub unsub (#751)
* feat: handle dynamic peer join and leave a pubSub topic

Co-authored-by: richΛrd <info@richardramos.me>


---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-09-19 11:35:29 +05:30
Prem Chaitanya Prathi dd5dc7a9c8
fix: change mutex to rwLock and fix relay unsubscribe (#749)
* fix: change mutex to rwLock and fix relay unsubscribe

* chore: modify relay test to cover unsubscribe and few more relay functions
2023-09-18 18:48:16 +05:30
Richard Ramos 2060c8c837 feat: allow running rln with no credentials 2023-09-15 10:18:44 -04:00
Richard Ramos 22b097817e feat(rln-relay): use zerokit 0.3.4 to obtain number of leaves set in merkle tree 2023-09-15 10:13:02 -04:00
Richard Ramos e1a84aab0e chore(rln-relay): docs and docker 2023-09-14 14:30:48 -04:00
Richard Ramos 55bc21c604 feat(rln-relay): isReady 2023-09-14 10:47:27 -04:00
Richard Ramos 7beaa3f029 feat(rln-relay): ensure execution order for pubsub validators 2023-09-14 09:38:28 -04:00
harsh jain 4b1c188cf0
feat: add common protocol design (#724)
* feat: add common protocol design

* fix: remove redundant vars

* fix: use AppDesign's ctx

* refactor: relay, add AppDesign

* feat: changes for suggestions

* test: commonService start/stop execution

* fix: lint error

* nit: add comments
2023-09-13 12:18:44 +07:00
Richard Ramos 2aea2f5ca5 fix: time calculation for fetching events 2023-09-12 12:09:30 -04:00
Richard Ramos 75486f8226 chore(rln-relay): add logs for fetching events 2023-09-12 11:22:07 -04:00
Richard Ramos dcc87cf24f chore(lint): adds `make lint-full` target and fixes some linting errors 2023-09-12 09:54:34 -04:00
Prem Chaitanya Prathi db25b307e2
Fix/filterv2 (#722)
* fix: updated filterv2 protocol as per rfc, make pubsub topic optional

* chore: make broadcaster optional in filter client

* reverting optional pubSub topic in Filter.MessagePush

* fix: revert pubSubTopic as optional and add higher level validation

* Update waku/v2/protocol/filter/client.go

Co-authored-by: richΛrd <info@richardramos.me>

---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-09-12 18:04:43 +05:30
Richard Ramos 18efd2c737 chore(rln-relay): clean up nullifier table every MaxEpochGap 2023-09-08 14:13:12 -04:00
Richard Ramos 793c059ea7 fix: loop counter for writing multiaddresses in enr 2023-09-08 13:45:11 -04:00
Richard Ramos 8783cd2f45 chore(rln-relay): use the only key from keystore if only 1 exists 2023-09-08 12:05:32 -04:00
harsh jain 08cabab41f
fix: and optimising fetching membership events (#706)
* fix: and optimising fetching membership events

* fix: start from lastProcessedBlock+1

* test: fetching membership logic

* refactor: usage of rlnInstance,rootTracker,groupManager

rlnInstance, rootTrack were previously created while creating rlnRelay
but were assigned to groupManager on Start of rlnRelay. This created
unncessary dependency of passing them to static and dynamic group
manager.
Web3Config uses interface EthClientI for client, so that we can pass
mock client for testing MembershipFetcher.

* fix: failing test

* fix: lint error

* fix: account for PR suggestions

* fix: failing race test

* fix: dont' increase fromBlock on error

* nit: fix naming and add comments
2023-09-07 23:23:48 +07:00
Richard Ramos def745cfba chore(rln-relay): confirm that the provided credential is correct using onchain query 2023-09-06 10:17:46 -04:00
Prem Chaitanya Prathi d13b1f0aa3
autosharding content-topic config (#696)
* chore: add shard choice simulation test

* feat: add new flags for pubsub and contentTopics and deprecate topic flag

* chore: remove store-resume-peer config and comment out functionality until redesign of store is done

* chore: fix code to use contentTopics value

* fix: use default waku topic only if no other topics are provided in the config
2023-09-06 10:07:21 +05:30
Richard Ramos f9179cd116 feat: store and retrieve valid merkle roots in RLN db 2023-09-04 10:51:40 -04:00
Richard Ramos f9ed8d973c fix(rln-relay): sync from deployed block number 2023-09-04 10:30:37 -04:00
Richard Ramos 84fa332e1c fix: allow mixing named and static shards 2023-09-04 10:10:49 -04:00
Richard Ramos 5fcfbb9897 feat: use rln registry contract 2023-09-04 10:02:14 -04:00
Richard Ramos ab3f21f209 chore: compile smart contracts and generate types 2023-09-04 10:02:14 -04:00
richΛrd 8aa1c4a39b
refactor(filter): unsubscribe waitgroup, execute async, and guard against calling functions while the protocol is not started (#692)
* refactor(filter): unsubscribe waitgroup and async
* refactor: verify started state for doing filter operations
2023-09-04 09:53:51 -04:00
Richard Ramos 9c0bebc859 refactor: use a map to store credentials instead of an array 2023-08-31 15:42:45 -04:00
Richard Ramos 0854edaf3d refactor: keystore as a data type 2023-08-31 15:42:45 -04:00
harsh jain 229fb7a970
refactor: peerManager test (#683)
* refactor: peerManager test

* fix: break down and fix peerManager test

* refactor: codeClimate linting for peerManager/ConnectionGater
2023-08-30 21:33:57 +07:00
harsh jain 4a546d12d3
fix: dont send to subscriptions that don't have any ContentTopic (#695) 2023-08-30 17:35:08 +07:00
richΛrd 4c52149fac
chore: log succesful message pushes (#694)
Co-authored-by: Prem Chaitanya Prathi <chaitanyaprem@gmail.com>
2023-08-30 10:48:06 +05:30
Andrea Maria Piana 8ad08d6b04
Check nils when deliting subscription (#691) 2023-08-29 12:53:39 +01:00
Richard Ramos 041dc4070a fix(filterV2): requestID and log request type 2023-08-25 13:05:04 -04:00
Prem Chaitanya Prathi 44d3ef6d78
feat: autosharding core logic (#669)
* feat: autosharding core logic
2023-08-25 09:55:38 +05:30
Richard Ramos cb3f5da322 feat: rln metrics 2023-08-24 11:26:37 -04:00
Richard Ramos ddb08adbbd chore: set limit to 5000 to not choke eth1 node and start rln before relay 2023-08-24 10:48:22 -04:00
Richard Ramos 624fe1bb2f chore(rln): run rln in all relay pubsubtopics + remove cli flags 2023-08-24 10:48:22 -04:00
Richard Ramos 5422af8130 fix(rln-relay): RLN DB should be aware of chain and contract address 2023-08-24 10:38:59 -04:00
Richard Ramos 7e36f91a5a fix(rln-relay): flush_interval incorrectly set 2023-08-24 10:38:28 -04:00
Richard Ramos f62ba67a92 chore: update smart contracts and documentation 2023-08-24 10:37:54 -04:00
Richard Ramos 0b943caaa8 refactor: remove credential registering from waku 2023-08-24 10:37:54 -04:00
Richard Ramos f088e49075 feat: create `generate-rln-credentials` subcommand 2023-08-24 10:37:54 -04:00
Richard Ramos 8cc92dfdef chore(rln-relay): rename keystore application to `waku-rln-relay` 2023-08-23 15:38:05 -04:00
Richard Ramos 2818c3d394 refactor: add libp2p metrics, and make each protocol responsable for defining its own metrics 2023-08-18 13:56:43 -04:00
richΛrd 387ee5f9ac
feat: merkle tree persistence
- use atomic operations API
- resume onchain sync from persisted tree db
- close eth client and db connection appropriately
- pass in the path to the tree db
- fix nwaku compatibility issues
2023-08-18 09:59:37 -04:00
Vitaliy Vlasov 419adcb6a8
chore(filter2): test updates 2023-08-14 16:29:00 -04:00
Richard Ramos c320b38cbe feat(rln-relay): use atomic operations API 2023-08-12 07:44:07 -04:00
Richard Ramos da45dab68e
fix: nwaku / go-waku rlnKeystore compat 2023-08-11 11:54:24 -04:00
Richard Ramos e0e4a2fa87 refactor: remove unused function and simplify code related to creating db and migrations 2023-08-10 10:37:22 -04:00
Richard Ramos ee2af4646c feat: sqlite vacuum and optional migrations 2023-08-10 10:29:40 -04:00
Prem Chaitanya Prathi 9f45d271ac
feat: support serviceslots in peermanager (#631)
* feat: support peermanager serviceslots and update store protocol to use serviceslots

* fix: lint errors in test code

* fix: error in nix build due to vendor sha change

* fix: set host in peermanager even if relay is disabled

* chore: fix codeclimate issues

* chore: using common filterPeer function to avoid duplication

* feat:use service slots in other service protocols

* chore: fix codeclimate issues

* chore: move AddPeer to peermanager

* Apply suggestions from code review

Co-authored-by: richΛrd <info@richardramos.me>

* chore:address review comments

* feat: implement RemovePeer #638

* chore: fix test failure

* Support for multiple slots for service peers
Adding discovered peers also moved to peer manager

---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-08-10 18:28:22 +05:30
Prem Chaitanya Prathi af7471d158
fix:during unsubscribeAll, remove peer without any checks 2023-08-07 15:01:44 +05:30
Prem Chaitanya Prathi 20040f2e9b
fix: data race in filter unsubscribe 2023-08-05 08:03:19 +05:30
Prem Chaitanya Prathi 22398b2868
fix:race in filter peer subscription map iteration and deletion 2023-08-05 08:00:04 +05:30
Prem Chaitanya Prathi 884f65ecdf
fix: noise protocol channel close race condition 2023-08-04 17:19:53 +05:30
Prem Chaitanya Prathi 34de2941c7
Feat/peer manager (#596)
* chore: refactor existing code into peer maanger package

* feat: move peer connection related code into peer manager

* feat: in relay peer connection pruning

* feat: add max-connections CLI flag and limit outRelayPeers based on max-connections #621

* tested both in and out relay connection limits

Co-authored-by: richΛrd <info@richardramos.me>

* Review comment, use context to pause connectivity loop during node shutdown.

Co-authored-by: richΛrd <info@richardramos.me>

* address review comments


---------

Co-authored-by: richΛrd <info@richardramos.me>
2023-08-03 21:51:15 +05:30
Richard Ramos 33344c2ae0 refactor: use uint instead of int for RLN indexes 2023-08-03 11:17:50 -04:00
Richard Ramos a4b038730c feat(rln-relay): pass in index to keystore credentials 2023-08-03 11:17:50 -04:00
Prem Chaitanya Prathi e6aaf9c852
chore: fix comment on test 2023-08-02 11:24:50 +05:30
Richard Ramos 55ec2347a8 fix: RLN
- subscription to topic is not required for registering RLN validator
- check for duplication msg within same epoch was incorrect
- added an example on how to use RLN
2023-08-01 09:19:38 -04:00
Richard Ramos 8b3f42310d fix: code review 2023-08-01 09:17:11 -04:00
Richard Ramos dc08c44804 feat: add filters for discv5 2023-07-31 10:45:45 -04:00
Richard Ramos a422add209 chore: add missing comments to functions 2023-07-23 07:01:42 -04:00
Richard Ramos dbd94ebb81 fix: add p2p-circuit component to multiaddresses that use circuit-relay 2023-07-07 12:04:05 -04:00
Richard Ramos 8cac406a4c fix: warning message format when using static shards and named shards at same time 2023-07-07 10:35:12 -04:00
Richard Ramos b26859fc6a refactor: make discovery connector subscribe to discovery services 2023-07-07 09:47:46 -04:00
Richard Ramos 6ece3c483b feat: subscribe to shards with --topic and refactors
- Shards in same cluster are advertised on ENR
- Store().Resume() was moved to app layer
- NoDefaultWakuTopic() was removed since it's the app that must determine whether it subscribes to the default waku topic
- Removed `Publish` from WakuNode (not really used and easy to implement in app layer if needed)
2023-07-06 11:18:29 -04:00
Richard Ramos 7dd02067f8 fix: unsubscribe from all peers unless specified and check err code when full node has no subscription 2023-07-05 15:54:31 -04:00
Richard Ramos fb3447f3d8
fix: adding relay protocol to feature check function of gossipsub 2023-06-29 09:29:08 -04:00
Richard Ramos b8ab99ba08 fix: set wakurelay at index 0 in gossipsub protocol list 2023-06-28 14:06:16 -04:00
Richard Ramos eba4aa43e5 feat: find discv5 peers with shards 2023-06-26 11:03:16 -04:00
Richard Ramos 0381b92531 feat(c-bindings): filterv2 2023-06-26 11:03:01 -04:00
Richard Ramos c380faa6ae fix: code review and disconnect bad peers 2023-06-09 14:57:43 -04:00
Richard Ramos 875d132c1e feat: gossipsub scoring 2023-06-09 14:57:43 -04:00
Richard Ramos 76186e5477 refactor: store ENR 2023-06-09 14:57:43 -04:00
Richard Ramos 52ac8e3740 refactor: various
- Limit inbound connections to 10 per IP
- Expose gossipsub parameters on WakuRelay
- New peerstore
2023-06-09 14:57:43 -04:00
Richard Ramos b0c094b0b6 feat: more utils functions to setup a relay shards field on the enr 2023-06-05 16:03:14 -04:00
Richard Ramos 1bffa35dfa fix: enodeToMultiaddress when using ipv6
Fixes #560
2023-06-01 14:02:18 -04:00
Richard Ramos 1c75c89790 refactor: remove unneeded `Connect` as dns4 addresses are being resolved now 2023-06-01 14:02:08 -04:00