195 Commits

Author SHA1 Message Date
Danish Arora
7c0ce7b2ec
chore: upgrade to libp2p v2 (#2143)
* chore: upgrade libp2p to v2 and related deps

* chore: fix ENR

* chore(core): remove CustomEvent polyfill import

* chore: `peer-id-factory` has been removed

* chore(discovery): fix local-cache & remove CustomEvent imports

* chore(sdk): update config

* chore(tests): update tests without peer-id-factory

* fix: spec tests

* chore: fix test

* chore: upgrade dataset-core

* chore: upgrade libp2p and stale references

* chore: upgrade playwright

* chore: rm console log

* fix: lock
2024-10-21 16:43:24 +05:30
Danish Arora
b2efce5ec2
feat(filter): enhancing protocol peer management with mutex locks (#2137)
* chore: improvements

* chore: add logs for subscription maintenance

* chore: update logging

* chore: trimming down BaseProtocolCore

* chore: track peers in a hashmap instead of array

* chore: peer mgmt responds to conenction/disconnection and improve logging

* feat: add mutex locks to tackle race conditions over shared state

* fix: build

* chore: some mutex lock-release improvements

* feat: peer manager

* chore: rm tests for remove internal util

* chore: update HealthManager updates

* chore: update tests

* rm: only

* fix: hasPeers management

* chore: add modularity to getting connected peers

* chore: improve logs & add debug

* chore: renewal doesnt disconnect, only removes

* chore: await for sequential operations

* chore: add TODO

* chore: minor improvements

* chore: fix rebase

* chore: update playright

* chore: remove additional arg

* chore: update interafce

* feat(peer-manager): unit tests

* chore: improve hasPeers()

* chore: update lockfile

* feat: Filter reacts to peer:disconnect event, add tests

* chore: fix lock

* chore: update playright

* chore: update protocol health for lightpush

* chore: remove .only

* chore: address comments and improvements

* fix: tsconfig
2024-10-10 23:47:12 +02:00
Sasha
1d68526e72
feat(lightPush): improve peer usage and improve readability (#2155)
* fix comment of default number of peers

* export default number of peers from base protocol sdk

* rename to light_push, move class to separate file

* move waitForRemotePeer to sdk package

* add todo to move waitForGossipSubPeerInMesh into @waku/relay

* clean up waitForRemotePeer, split metadata await from event and optimise, decouple from protocol implementations

* simplify and rename ILightPush interface

* use only connected peers in light push based on connections instead of peer renewal mechanism

* improve readability of result processing in light push

* fix check & update tests

* address tests, add new test cases, fix racing condition in StreamManager

* use libp2p.getPeers
2024-10-04 10:50:58 +02:00
Sasha
b93134a517
fix: peer renewal connection drop & stream management (#2145)
* fix: peer renewal connection drop

* fix stream manager

* fix over iteration during stream creation

* remove timeout and use only open peers

* add logs

* refactor code, add tests

* debug test

* up debug

* remove debug, supress check for timestamps

* remove only

* add more debug

* remove debug

* remove check for timestamps
2024-10-01 12:54:55 +02:00
Sasha
0263cb80c5
fix: improve node bootstrapping (#2121)
* fix: improve node bootstrapping

* implement dns discovery on failed dials

* update const to match RFC

* up
2024-09-06 01:10:54 +02:00
Sasha
a022433851
feat(lightpush)!: return new error messages (#2115)
* feat: return new error messages

* fix test, remove unused

* up

* up

* rollback

* up test
2024-08-29 11:20:19 +02:00
Sasha
eadb85ab83
feat: offline state recovery for Filter subscription (#2049)
* up

* fix window reference

* add tests

* up

* add e2e renew test

* address comments

* remove unused

* add test

* try

* remove only

* up test

* up

* remove only

* add tmp logs, use before/after hooks

* up

* fix check

* remove only

* fix test

* up
2024-08-28 18:00:18 +02:00
Danish Arora
5ce36c8f18
feat!: deprecate named pubsub topics and use static/auto sharding (#2097)
* feat: deprecate named sharding & protocols adhere
simplify network config type, all protocols use pubsub topic internally

* chore: update tests

* tests: rm application info

* chore: use static sharding and auto sharding terminologies

* chore: update docs for network config

* chore: update interfaces

* tests: update tests error message

* chore: remove `ShardingParams` type and fix test
2024-08-13 05:23:20 +05:30
Danish Arora
86f730f958
feat!: store v3 (#2036)
* feat: write proto

* chore: move store v2 to a subdir

* chore: update v3 proto

* feat: create custom RPC

* feat: implement storev3 core

* chore: set store v3 as default

* chore: move v2 related code

* chore: update v2 imports

* feat: add store-v3 sdk implementation

* fix: rebase

* chore: add ts-doc for store query request params

* chore: update tests for new API

* fix: use nanoseconds instead of millisecond for timerange

* chore: improve store

* chore: remove store v2

* chore: update tests

* chore: fix legacy imports & proto

* tests: remove manual reversal as its part of the API, update incorrect cursor error msg

* chore: update default page size

* chore: account for MAX_PAGE_SIZE from nwaku

* fix: test

* fix: sorting tests
2024-08-06 12:06:37 +05:30
Danish Arora
d464af3645
feat: node and protocols health (#2080)
* feat: introduce HealthManager

* feat: make health accessible on Waku object

* feat: update health from protocols

* chore: add access modifiers to healthmanager

* feat: use a HealthManager singleton

* chore: add tests for Filter, LightPush and Store

* feat: add overall node health

* chore: update protocol health to consider Store protocol

* chore: setup generic test utils instead of using filter utils

* tests: add a health status matrix check from 0-3

* chore: increase timeout for failing tests in CI
tests pass locally without an increased timeout, but fail in CI

* chore: move name inference to HealthManager

* tests: abstract away node creation and teardown utils

* fix: import
2024-07-27 18:27:54 +05:30
Danish Arora
169a09d552
chore: enforce access modifiers (#2068)
* feat: introduce eslint flag

* chore: update logger

* chore: update enr

* chore: update core

* chore: update sdk

* chore: update relay

* chore: update discovery

* chore: update message-encryption

* chore: update tests

* chore: fix modifiers

* chore(tests): fix access modifiers

* chore: fix rebase
2024-07-19 15:58:17 +05:30
Danish Arora
9b0f1e855a
feat: validate messages for individual filter nodes & perform renewals (#2057)
* feat: validate messages for individual filter nodes & perform renewals

* chore: fix spell check

* chore: use a max threshold before peer renewal

* chore: switch from a validation cycle timer to adhoc validation

* chore: add test

* fix: test

* chore: address comments

* fix: renewal without a new peer available

* chore: validating messages should be non-blocking

* chore: minor improvements

* chore: rm only

* chore: fix test
2024-07-18 00:41:21 +05:30
Sasha
00635b7afe
feat: fix peer renewal, change Filter keep alive (#2065)
* move util, create stream manager folder

* change default keep alive, improve peer renewal

* address nit
2024-07-16 18:35:24 +02:00
Danish Arora
93e78c3b87
feat(lightpush): peer management for protocols (#2003)
* chore: make `dropConnection` to be a public function

* feat: peers are maintained for protocols
- passes `ConnectionManager` to ProtocolSDK to allow disconnecting from within protocol
- maintains `numPeersToUse` for each protocol within BaseProtocolSDK

* fix: pass options to protocols

* chore: update interfaces to allow public access

* chore: improve logging on protocol

* fix: renew peer upon failure

* chore(tests): allow DefaultPubsubTopic

* feat(lightpush): write peer management tests

* chore: rename test

* feat: add lock to `maintainPeers()` to handle parallelisation of requests
fixes parallelisation of lightpush.send() requests

* fix: concurrent lightpush requests

* fix: test & improve peers fetching

* chore: use getter

* address comments

* chore: smaller improvements

* feat: attempt to improve time for first lightpush.send()

* chore: use `window.interval` for type-safety

* chore: remove delays

* feat: add autoRetry

* feat: `forceUseAllPeers` to wait for all connected peers to be resoled
2024-06-19 01:52:16 -04:00
Danish Arora
984fb94b5b
fix: failing node_optional check (#2025)
* chore: remove predefine nodes list

* remove import

* chore: update & fix test

* chore: remove console logs
2024-06-18 13:43:22 -04:00
gabrielmer
2311a595b2
fix: increasing maxInboundStreams for lightpush from 32 to 100 (#2021)
* increasing maxInboundStreams for lightpush from 32 to 100

* allowing test to fail
2024-05-30 13:58:50 +05:30
Danish Arora
e49e7289ae
chore: improve StreamManager (#1994)
* chore: improve stream manager

* chore: some cleaning

* chore: update API

* chore: rename `getConnectionStatus` to `isConnected`

* chore: use throw/catch

* chore: remove await

* remove redundant function and rename

* chore: increase test timeout
2024-05-14 16:31:38 +05:30
Danish Arora
4eb06c64eb
feat(filter)!: return error codes instead of throwing errors (#1971)
* move protocol result type to interfaces

* chore: update type names for verbosity

* feat(filter-core): convert error throws to return types

* chore: update types & imports

* update Filter API

* chore: update createSubscription

* chore: update imports & rename

* chore: update all tests

* chore: resolve conflicts & merge (2/n)

* chore: resolve conflicts & merge (3/n)

* chore: resolve conflicts & merge (4/n)

* chore: resolve conflicts & merge (5/n)

* chore: resolve conflicts & merge (6/n)

* chore: use idiomatic approach

* chore: fix tests

* chore: address comments

* chore: fix test

* rm: only
2024-05-09 16:51:08 +05:30
Arseniy Klempner
fb34b7262a
fix: only override ping metadata in peer store (#1984)
Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2024-04-30 00:39:02 +02:00
Sasha
f3627c46a4
feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default (#1958)
* fix: use pubsubTopic from current ones if not set

* fix: improve type on dial method

* enforce same pubusb on filter.subscribe, make content topic to pubsub mapping default for decoder / encoder

* fix mapping problem

* update tests

* add error handling

* fix typo

* up lock

* rm lock

* up lock

* remove only

* fix content topic

* fix ephemeral test

* fix filter unsubscribe test

* up utils

* fix subscribe test

* up interfaces and filter api

* remove only

* up ping test

* fix subscribe test

* fix push test

* fix lightPush

* fix multiple pubsub

* remove only, fix subscribe filter test

* remove only

* fix cluster ID selection and named sharding subscription test

* fix unsubscribe test

* fix light push test

* fix light push test

* fix push test

* fix relay publish

* create runNode and fix relay tests

* generalize runNodes, fix some tests

* fix store tests

* fix toAsyncIterator tests

* remove only

* fix lightPush

* use generics

* try fix test

* run failing tests

* remove only

* address failed tests, remove DefaultPubsubTopic dependency in some tests
2024-04-28 11:15:17 +02:00
Arseniy Klempner
9514653428
fix: use correct shard index when creating encoder 2024-04-24 18:40:17 -07:00
Danish Arora
5fb100602b
chore(lightpush)!: move protocol implementation to @waku/sdk (1/n) (#1964)
* chore: decouple `Filter` between `core` and `sdk`
moves `SubscriptionManager` to `sdk` side

* chore: update package dependencies
also update peer deps in sdk

* chore: update imports

* chore: update tests

* chore(side-change): update lightpush

* chore: update size-limit import

* chore(sdk): update dependencies
2024-04-19 17:20:34 +05:30
Anton Iakimov
9f2c0ed6b5
chore: switch wakuv2 fleet to waku (#1910)
* chore: switch from decomissioned wakuv2 fleet

* chore: minor fixes + CI

* chore: fix eslint

* fix: eslint

* chore: fix test

---------

Co-authored-by: danisharora099 <danisharora099@gmail.com>
2024-04-07 15:21:00 +03:00
Danish Arora
bf42c8f53a
chore(store)!: move protocol implementation opinions to @waku/sdk (#1913)
* refactor the Store protocol into Core and SDK, simplify `queryGenerator()`

* update imports & types

* chore: `@noble/hashes` moves to `sdk`

* chore: update tests

* chore: update size-limit import path

* fix: cursor tests, use `Cursor` type from `proto.Index` instead of redefining

* export wakuStore from sdk

* fix: imports

* chore: use specific version for package

* chore: handle error for peer access

* use type instead of interface

* rm: comment

* add TODO

* chore!: remove deprecated function definition

* chore: improve logging
2024-04-01 16:47:47 +05:30
Danish Arora
877fe1dc1d
feat: peer-exchange uses error codes (#1907)
* setup a generic protocol result type (DRY)

* metadata: use generic

* lightpush: use generic

* peer-exchange: use error codes + generic + update tests

* add issue link to skipped test

* tests: improve while loop readability
2024-03-13 19:33:50 +05:30
Danish Arora
1882023c58
feat(metadata): use error codes (#1904) 2024-03-12 16:40:08 +05:30
Danish Arora
8deab11890
chore(lightpush)!: move protocol implementation opinions to @waku/sdk (#1887)
* chore: restructure @waku/sdk

* chore: introduce `BaseProtocolCore` and `BaseProtocolSDK`

* chore: introduce `LightPushCore` and `LightPushSDK`

* chore: update `relay` for new types

* chore(sdk): update structure

* chore(filter): add `numPeersToUse`

* chore: update tests

* update: size-limit

* chore: update more tests

* attach issue link to TODOs
2024-03-11 18:50:34 +05:30
Arseniy Klempner
ee2d4176f8
feat: create node and subscription by content topic 2024-02-27 15:37:57 -08:00
Danish Arora
199f6ab2ff
feat: local discovery (#1811)
* initialise the new package

* feat: implement LocalStorageDiscovery

* add: Startable to cspell

* add compliance test

* add: discovery tests

* rm: browser tests script

* address comments

* add type safety to getting peers from local

* only dispatch peer if it does not exist

* move ws ma extraction to utils

* chore: update package name to local-discovery

* fix: add compliance test with no external deps on service node

* use peer:identify instead of peer:update

* add: unit tests & remove sdk dependency

* move tests to self package

* update cspell + remove unrequired deps

* add types

* maintain in-memory peers for localstorage

* address comments

* chore: rename

* use name from options

* fix: saving peers

* rm: only
2024-02-16 20:06:27 +05:30
Arseniy Klempner
e138b4f5c4
feat: decouple sharding params out of core 2024-02-08 08:42:47 -08:00
Arseniy Klempner
68d3229644
feat: make ShardingParams optional in sdk, required internally 2024-01-24 07:36:00 -08:00
Danish Arora
7affbe222d
feat: lightpush & filter send requests to multiple peers (#1779)
* feat: lightpush & filter send requests to multiple peers

* build message-hash before core

* chore: restructure folder heirrarchy

* fix: imports

* chore: move @waku/core to dev deps

* feat: create a new `ServiceNodes` wrapper class to encapsulate service node and message collector with redundancy accounted for

* chore(filter): move tests against single service node to a subdir

* feat: support relay, add strict checking, add tests

* fix(filter): handle errors

* chore(tests): add tests for ping

* add tests for push

* chore: abstract redundancy

* feat: add unsubscribe tests

* fix: tests

* add lightpush tests

* fix: imports

* fix: merge & add warning

* merge: master

* fix: breaking tests with master

* address comments

* make num peers configurable

* fix: typo
2024-01-24 18:24:03 +05:30
Danish Arora
477c2a5918
feat!: protocols filter peers as per configured shard (#1756)
* merge: master

* fix: tests

* update: interfafces

* rm: comments

* metadata: store peerIdStr instead of peerId

* chore(utils): move fast-utils to dev deps

* fix: allow autosharding nodes to get peers (#1785)

* fix: merge

* fix: build

* fix: failing tests from master merge

---------

Co-authored-by: Arseniy Klempner <arseniyk@status.im>
2024-01-19 20:42:52 +05:30
Sasha
ea4b30752d
chore: fix nwaku interop test failures (#1792)
* chore: investigate interop test failures

* try some fixes

* remove only

* remove comment

* use getConnections for waku node
2024-01-18 11:05:35 +01:00
Arseniy Klempner
ae7bc3eeec
chore: bump all libp2p packages to latest version 2024-01-17 11:59:08 -08:00
Danish Arora
b99f828cfb
chore: minor refactoring for protocols (#1762)
* remove: unnecessary function

* remove: test

* update doc

* add: tests

* revert unintended change

* fix: ping return

* fix(tests): stub connections

* log warning instead of throwing an error
2024-01-11 17:25:47 +05:30
Arseniy Klempner
69406bf1f0
reintroduce and deprecate named sharding (#1751)
Co-authored-by: danisharora099 <danisharora099@gmail.com>
2024-01-10 13:04:30 +05:30
Danish Arora
9983549e8b
chore: use connected peers instead of discovered peers for protocols (#1758)
* switch to use connected peers instead of discovered peers for protocols

* switch to use connected peers instead of discovered peers for protocols
2024-01-02 15:49:31 +05:30
danisharora099
2bc3735e4d
feat: add support for autosharded pubsub topics
tests: use a generator for sharded pubsub topics
set pubsub topic in encoder/decoder based on sharding type
add function for grouping content topics by pubsub topic
add autosharding config to create options
add autoshard rpc endpoints to nwaku and use in tests
set autoshard pubsub topics in all protocols
fix rebase with static sharding
removes unused function
remove console logs
remove autosharding from ShardInfo, add to EncoderOptions
fix enr and encoder/decoder options
test that same application/version hashes to same shard index
update comment on shard field
fix spelling of autosharding
fix content topic protocol in tests
add sharding type alias and function to determine topic in encoders/decoders
move DefaultPubsubTopic from core to interfaces
2023-12-21 10:03:22 -08:00
Danish Arora
9ac2a3f363
feat: metadata protocol (#1732)
* add proto

* add rpc and interfaces

* add protocol implementation

* update faulty proto def

* add rpc and interfaces

* refactor implementation & write test

* setup the metadata protocol as a service

* fix cases where metadata service needs to be undefined

* remove redundant catch block

* remove addressed TODO

* update import path

* log errors

* remove redundant code from handling incoming metadata request

* update tests

* add test to check for active connections

* change expects

* save remote peer's shard info after successful connection
2023-12-05 19:26:52 +05:30
Danish Arora
7ce642c2cc
chore: fix encoder/decoder creation (#1742)
* fix encoder creation

* bump nwaku to v0.22.0 (#1741)
2023-11-29 17:30:56 +05:30
Danish Arora
4cf2ffefa7
feat!: add support for sharded pubsub topics & remove support for named pubsub topics (#1697)
* merge branches

* tests: use a generator for sharded pubsub topics

* fix namespace edge case

* move shardInfo to pubsubTopic logic in waku.ts

* simplify encoder/decoder creation logic + update tests

* sharding utils: add error handling

* remove redundant test util

* baseprotocol: create abstraction for initialising pubsub topics

* fix: `createDecoder` interface

* filter: createSubscription takes shardInfo instead of pubsubTopicStr

* fix: sharding utils for error handling

* SingleShardInfo: use a new interface instead of omitting and rename namespace

* change redundant namespaces
2023-11-28 15:57:18 +05:30
Arseniy Klempner
1d0e2ace7f
feat: track node connection state (#1719)
Co-authored-by: chair <29414216+chair28980@users.noreply.github.com>
Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2023-11-27 12:44:49 +01:00
Danish Arora
3166a5135e
chore!: change all instances of PubSubTopic to PubsubTopic (#1703)
* rename all PubSub patterns

* feat: forbid identifiers with camelcase pubSub (#1709)

---------

Co-authored-by: Arseniy Klempner <adklempner@gmail.com>
2023-11-14 18:52:52 +03:00
fryorcraken
b7dc3d7576
fix: measure total message size (#1643)
Network message limitations are imposed on the whole message, not just
the payload.

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2023-10-26 13:14:47 +02:00
Vaclav Pavlin
b3864f8772
fix: handle all empty responses in filter (#1688) 2023-10-26 10:46:08 +02:00
Danish Arora
491366bd4f
fix: filter subscription with pubsubTopic1 and decoder with pubsubTopic2 (#1675)
* add a test for a failing case

* add error handling & update test

* remove only

* rename test
2023-10-23 14:17:02 +05:30
Danish Arora
81b2bf46ab
allow filtering based on log levels (#1677) 2023-10-23 13:26:35 +05:30
Danish Arora
1892f5093d
fix: don't dial discovered peers if have already been attempted dial (#1657)
* don't dial peers if they exist in PeerStore already

* fix(tests): bugs & add types to dispatch event

* fix: more tests

* fix: dial validation

* update doc & reduce delay

* refactor test

* use -1 instead of Infinity

* fix comment

* fix rebase
2023-10-20 18:16:48 +05:30
Danish Arora
0f7d63ef93
feat: Logger with log levels (#1672)
* setup a custom Logger with log level support

* refactor codebase for to use new Logger with log levels

* disallow usage of `debug` directly / only allow usage in/through custom Logger

* remove `debug` from logger
2023-10-20 16:36:47 +05:30