Commit Graph

183 Commits

Author SHA1 Message Date
Danish Arora 409642d7d3
chore: upgrade libp2p and related packages (#2084)
* chore: upgrade libp2p and related packages

* fix: public key on peer id can now be undefined

* chore: upgrade remainder packages

* chore: fix tests

* chore: address comments

* chore: upgrade playwright and proto
2024-08-14 04:23:04 +05:30
Danish Arora 5ce36c8f18
feat!: deprecate named pubsub topics and use static/auto sharding (#2097)
* feat: deprecate named sharding & protocols adhere
simplify network config type, all protocols use pubsub topic internally

* chore: update tests

* tests: rm application info

* chore: use static sharding and auto sharding terminologies

* chore: update docs for network config

* chore: update interfaces

* tests: update tests error message

* chore: remove `ShardingParams` type and fix test
2024-08-13 05:23:20 +05:30
Danish Arora 86f730f958
feat!: store v3 (#2036)
* feat: write proto

* chore: move store v2 to a subdir

* chore: update v3 proto

* feat: create custom RPC

* feat: implement storev3 core

* chore: set store v3 as default

* chore: move v2 related code

* chore: update v2 imports

* feat: add store-v3 sdk implementation

* fix: rebase

* chore: add ts-doc for store query request params

* chore: update tests for new API

* fix: use nanoseconds instead of millisecond for timerange

* chore: improve store

* chore: remove store v2

* chore: update tests

* chore: fix legacy imports & proto

* tests: remove manual reversal as its part of the API, update incorrect cursor error msg

* chore: update default page size

* chore: account for MAX_PAGE_SIZE from nwaku

* fix: test

* fix: sorting tests
2024-08-06 12:06:37 +05:30
Danish Arora fdd9dc44a4
feat(filter)!: new simpler filter API (#2092)
* chore: rename IReceiver subscribe

* feat!: new `subscribe() API that only takes in decoders and callback

* chore: `to_async_iterator` uses new function name

* chore: make `createSubscription` private, and shorten error handling

* chore: update subscribe return type

* tests: use new API

* fix: tests
2024-08-05 15:52:58 +05:30
Danish Arora d464af3645
feat: node and protocols health (#2080)
* feat: introduce HealthManager

* feat: make health accessible on Waku object

* feat: update health from protocols

* chore: add access modifiers to healthmanager

* feat: use a HealthManager singleton

* chore: add tests for Filter, LightPush and Store

* feat: add overall node health

* chore: update protocol health to consider Store protocol

* chore: setup generic test utils instead of using filter utils

* tests: add a health status matrix check from 0-3

* chore: increase timeout for failing tests in CI
tests pass locally without an increased timeout, but fail in CI

* chore: move name inference to HealthManager

* tests: abstract away node creation and teardown utils

* fix: import
2024-07-27 18:27:54 +05:30
gabrielmer 08fc2d133a
chore: changing default pubsub topic to its static sharding version (#2083)
* changing default pubsub topic to its static sharding version

* keeping RFC's Waku Message test vectors

* reverting change in changelog

* setting pubsub topic when creating nwaku node

* adding shardInfo to runMultipleNodes call

* adding shardInfo to runMultipleNodes call in lightpush tests

* add pubsub topics to nwaku.start

* get rid of it.only that remained

* fixing compliance tests

* setting clusterId to 0

* removing unnecessary fix

* adding shardInfo when creating nodes

* fixing wait for remote peer tests

* fixing peer exchange test

* refactor

* removing unnecessary variable

* feat: create default shard info, update tests (#2085)

* feat: create default shard info, update tests

* add link

* fix tests

* remoe only

* up tests

* up test

---------

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2024-07-26 00:21:52 +02:00
Danish Arora 9b0f1e855a
feat: validate messages for individual filter nodes & perform renewals (#2057)
* feat: validate messages for individual filter nodes & perform renewals

* chore: fix spell check

* chore: use a max threshold before peer renewal

* chore: switch from a validation cycle timer to adhoc validation

* chore: add test

* fix: test

* chore: address comments

* fix: renewal without a new peer available

* chore: validating messages should be non-blocking

* chore: minor improvements

* chore: rm only

* chore: fix test
2024-07-18 00:41:21 +05:30
Sasha ccdf84e593
chore: release master (#2004)
reliability improvements
2024-07-10 17:25:22 +02:00
Danish Arora 318667e442
feat(filter): peer/subscription renewal with recurring Filter pings (#2052)
* chore: renewPeer() returns the new found peer

* feat: ping & peer renewal

* chore: add tests

* fix: tests

* chore: remove only

* chore: remove comments

* chore(tests): decrease timeout

* chore: add array index validation

* chore: remove only

* chore: move defaults into a separate variable

* chore: update lightpush with new API

* chore: include peer renewals within `ping` instead of `interval`

* chore: update tests

* chore: add new test

* chore: address comments
2024-07-10 15:34:16 +05:30
Danish Arora 4db508b962
feat(filter): use protocol peer management (#2047)
* feat: leverage protocol peer management

* chore: add test

* chore: address comments

* chore: add todo
2024-07-03 12:09:34 +05:30
Danish Arora 93e78c3b87
feat(lightpush): peer management for protocols (#2003)
* chore: make `dropConnection` to be a public function

* feat: peers are maintained for protocols
- passes `ConnectionManager` to ProtocolSDK to allow disconnecting from within protocol
- maintains `numPeersToUse` for each protocol within BaseProtocolSDK

* fix: pass options to protocols

* chore: update interfaces to allow public access

* chore: improve logging on protocol

* fix: renew peer upon failure

* chore(tests): allow DefaultPubsubTopic

* feat(lightpush): write peer management tests

* chore: rename test

* feat: add lock to `maintainPeers()` to handle parallelisation of requests
fixes parallelisation of lightpush.send() requests

* fix: concurrent lightpush requests

* fix: test & improve peers fetching

* chore: use getter

* address comments

* chore: smaller improvements

* feat: attempt to improve time for first lightpush.send()

* chore: use `window.interval` for type-safety

* chore: remove delays

* feat: add autoRetry

* feat: `forceUseAllPeers` to wait for all connected peers to be resoled
2024-06-19 01:52:16 -04:00
Danish Arora e49e7289ae
chore: improve StreamManager (#1994)
* chore: improve stream manager

* chore: some cleaning

* chore: update API

* chore: rename `getConnectionStatus` to `isConnected`

* chore: use throw/catch

* chore: remove await

* remove redundant function and rename

* chore: increase test timeout
2024-05-14 16:31:38 +05:30
Danish Arora 4eb06c64eb
feat(filter)!: return error codes instead of throwing errors (#1971)
* move protocol result type to interfaces

* chore: update type names for verbosity

* feat(filter-core): convert error throws to return types

* chore: update types & imports

* update Filter API

* chore: update createSubscription

* chore: update imports & rename

* chore: update all tests

* chore: resolve conflicts & merge (2/n)

* chore: resolve conflicts & merge (3/n)

* chore: resolve conflicts & merge (4/n)

* chore: resolve conflicts & merge (5/n)

* chore: resolve conflicts & merge (6/n)

* chore: use idiomatic approach

* chore: fix tests

* chore: address comments

* chore: fix test

* rm: only
2024-05-09 16:51:08 +05:30
Sasha 3d92f194a7
chore: release master (#1957) 2024-04-30 17:50:00 +02:00
Sasha 5b03709dfe
feat: lift contentTopics and make shardInfo mandatory for createLight… (#1959)
* feat: lift contentTopics and make shardInfo mandatory for createLightNode

* add default shard info and wanring

* fix

* fix lint

* fix

* fix types

* remove breaking changes
2024-04-30 01:47:45 +02:00
Sasha 1a6bc4f8ce
feat: add keep alive to Filter (#1970)
* fix: use pubsubTopic from current ones if not set

* fix: improve type on dial method

* enforce same pubusb on filter.subscribe, make content topic to pubsub mapping default for decoder / encoder

* fix mapping problem

* update tests

* add error handling

* fix typo

* up lock

* rm lock

* up lock

* remove only

* feat: implement keep alive for filter subscription

* remove

* address comments
2024-04-29 23:31:09 +02:00
Arseniy Klempner fa523b78af
feat: add libp2p option for max ping connections 2024-04-29 08:21:37 -07:00
Sasha f3627c46a4
feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default (#1958)
* fix: use pubsubTopic from current ones if not set

* fix: improve type on dial method

* enforce same pubusb on filter.subscribe, make content topic to pubsub mapping default for decoder / encoder

* fix mapping problem

* update tests

* add error handling

* fix typo

* up lock

* rm lock

* up lock

* remove only

* fix content topic

* fix ephemeral test

* fix filter unsubscribe test

* up utils

* fix subscribe test

* up interfaces and filter api

* remove only

* up ping test

* fix subscribe test

* fix push test

* fix lightPush

* fix multiple pubsub

* remove only, fix subscribe filter test

* remove only

* fix cluster ID selection and named sharding subscription test

* fix unsubscribe test

* fix light push test

* fix light push test

* fix push test

* fix relay publish

* create runNode and fix relay tests

* generalize runNodes, fix some tests

* fix store tests

* fix toAsyncIterator tests

* remove only

* fix lightPush

* use generics

* try fix test

* run failing tests

* remove only

* address failed tests, remove DefaultPubsubTopic dependency in some tests
2024-04-28 11:15:17 +02:00
Arseniy Klempner 9514653428
fix: use correct shard index when creating encoder 2024-04-24 18:40:17 -07:00
Danish Arora 5fb100602b
chore(lightpush)!: move protocol implementation to `@waku/sdk` (1/n) (#1964)
* chore: decouple `Filter` between `core` and `sdk`
moves `SubscriptionManager` to `sdk` side

* chore: update package dependencies
also update peer deps in sdk

* chore: update imports

* chore: update tests

* chore(side-change): update lightpush

* chore: update size-limit import

* chore(sdk): update dependencies
2024-04-19 17:20:34 +05:30
Sasha 01a99ce33c
chore: release master (#1950) 2024-04-09 14:11:20 +03:00
dependabot[bot] 0b54d2e3d7
chore(deps)(deps-dev): bump cspell from 8.6.0 to 8.6.1 (#1930)
Bumps [cspell](https://github.com/streetsidesoftware/cspell/tree/HEAD/packages/cspell) from 8.6.0 to 8.6.1.
- [Release notes](https://github.com/streetsidesoftware/cspell/releases)
- [Changelog](https://github.com/streetsidesoftware/cspell/blob/main/packages/cspell/CHANGELOG.md)
- [Commits](https://github.com/streetsidesoftware/cspell/commits/v8.6.1/packages/cspell)

---
updated-dependencies:
- dependency-name: cspell
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-04 16:42:51 +03:00
Danish Arora bf42c8f53a
chore(store)!: move protocol implementation opinions to `@waku/sdk` (#1913)
* refactor the Store protocol into Core and SDK, simplify `queryGenerator()`

* update imports & types

* chore: `@noble/hashes` moves to `sdk`

* chore: update tests

* chore: update size-limit import path

* fix: cursor tests, use `Cursor` type from `proto.Index` instead of redefining

* export wakuStore from sdk

* fix: imports

* chore: use specific version for package

* chore: handle error for peer access

* use type instead of interface

* rm: comment

* add TODO

* chore!: remove deprecated function definition

* chore: improve logging
2024-04-01 16:47:47 +05:30
dependabot[bot] cb8921e4a2
chore(deps)(deps-dev): bump cspell from 8.3.2 to 8.6.0 (#1927)
Bumps [cspell](https://github.com/streetsidesoftware/cspell) from 8.3.2 to 8.6.0.
- [Release notes](https://github.com/streetsidesoftware/cspell/releases)
- [Changelog](https://github.com/streetsidesoftware/cspell/blob/main/CHANGELOG.md)
- [Commits](https://github.com/streetsidesoftware/cspell/compare/v8.3.2...v8.6.0)

---
updated-dependencies:
- dependency-name: cspell
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-03-22 23:50:36 +01:00
Danish Arora 877fe1dc1d
feat: peer-exchange uses error codes (#1907)
* setup a generic protocol result type (DRY)

* metadata: use generic

* lightpush: use generic

* peer-exchange: use error codes + generic + update tests

* add issue link to skipped test

* tests: improve while loop readability
2024-03-13 19:33:50 +05:30
Danish Arora 1882023c58
feat(metadata): use error codes (#1904) 2024-03-12 16:40:08 +05:30
Danish Arora 8deab11890
chore(lightpush)!: move protocol implementation opinions to `@waku/sdk` (#1887)
* chore: restructure @waku/sdk

* chore: introduce `BaseProtocolCore` and `BaseProtocolSDK`

* chore: introduce `LightPushCore` and `LightPushSDK`

* chore: update `relay` for new types

* chore(sdk): update structure

* chore(filter): add `numPeersToUse`

* chore: update tests

* update: size-limit

* chore: update more tests

* attach issue link to TODOs
2024-03-11 18:50:34 +05:30
Sasha 8eb5703a27
chore: release master (#1881) 2024-03-05 00:11:44 +01:00
Sasha 9f198dd149
feat: add bootstrapPeers option and refactor sdk (#1871)
* move relay related code

* move libp2p to utils

* define CreateWakuNodeOptions

* improve options

* make libp2p use from create function

* add bootstrapPeers option

* fix lint

* fix types, imports

* fix exports

* use bootstrap along default bootstrap

* fix test as REST does not return peer though connection is made

* rollback condition change

* enable gossipSub back for every node
2024-03-04 10:56:20 +01:00
Danish Arora f3cb10d484
chore!: rename local-discovery to local-peer-cache-discovery (#1867) 2024-02-29 01:05:54 +05:30
Arseniy Klempner ee2d4176f8
feat: create node and subscription by content topic 2024-02-27 15:37:57 -08:00
dependabot[bot] 6d3ed304a8
chore(deps)(deps): bump @chainsafe/libp2p-gossipsub (#1866)
Bumps [@chainsafe/libp2p-gossipsub](https://github.com/ChainSafe/js-libp2p-gossipsub) from 11.1.0 to 12.0.0.
- [Release notes](https://github.com/ChainSafe/js-libp2p-gossipsub/releases)
- [Changelog](https://github.com/ChainSafe/js-libp2p-gossipsub/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ChainSafe/js-libp2p-gossipsub/compare/v11.1.0...v12.0.0)

---
updated-dependencies:
- dependency-name: "@chainsafe/libp2p-gossipsub"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-27 22:01:02 +05:30
Danish Arora 199f6ab2ff
feat: local discovery (#1811)
* initialise the new package

* feat: implement LocalStorageDiscovery

* add: Startable to cspell

* add compliance test

* add: discovery tests

* rm: browser tests script

* address comments

* add type safety to getting peers from local

* only dispatch peer if it does not exist

* move ws ma extraction to utils

* chore: update package name to local-discovery

* fix: add compliance test with no external deps on service node

* use peer:identify instead of peer:update

* add: unit tests & remove sdk dependency

* move tests to self package

* update cspell + remove unrequired deps

* add types

* maintain in-memory peers for localstorage

* address comments

* chore: rename

* use name from options

* fix: saving peers

* rm: only
2024-02-16 20:06:27 +05:30
Arseniy Klempner e138b4f5c4
feat: decouple sharding params out of core 2024-02-08 08:42:47 -08:00
dependabot[bot] 749d84d5e4
chore(deps)(deps-dev): bump cspell from 7.3.6 to 8.3.2 (#1825)
Bumps [cspell](https://github.com/streetsidesoftware/cspell) from 7.3.6 to 8.3.2.
- [Release notes](https://github.com/streetsidesoftware/cspell/releases)
- [Changelog](https://github.com/streetsidesoftware/cspell/blob/main/CHANGELOG.md)
- [Commits](https://github.com/streetsidesoftware/cspell/compare/v7.3.6...v8.3.2)

---
updated-dependencies:
- dependency-name: cspell
  dependency-type: direct:development
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-05 22:57:20 +01:00
Arseniy Klempner 68d3229644
feat: make ShardingParams optional in sdk, required internally 2024-01-24 07:36:00 -08:00
Danish Arora 7affbe222d
feat: lightpush & filter send requests to multiple peers (#1779)
* feat: lightpush & filter send requests to multiple peers

* build message-hash before core

* chore: restructure folder heirrarchy

* fix: imports

* chore: move @waku/core to dev deps

* feat: create a new `ServiceNodes` wrapper class to encapsulate service node and message collector with redundancy accounted for

* chore(filter): move tests against single service node to a subdir

* feat: support relay, add strict checking, add tests

* fix(filter): handle errors

* chore(tests): add tests for ping

* add tests for push

* chore: abstract redundancy

* feat: add unsubscribe tests

* fix: tests

* add lightpush tests

* fix: imports

* fix: merge & add warning

* merge: master

* fix: breaking tests with master

* address comments

* make num peers configurable

* fix: typo
2024-01-24 18:24:03 +05:30
Danish Arora 477c2a5918
feat!: protocols filter peers as per configured shard (#1756)
* merge: master

* fix: tests

* update: interfafces

* rm: comments

* metadata: store peerIdStr instead of peerId

* chore(utils): move fast-utils to dev deps

* fix: allow autosharding nodes to get peers (#1785)

* fix: merge

* fix: build

* fix: failing tests from master merge

---------

Co-authored-by: Arseniy Klempner <arseniyk@status.im>
2024-01-19 20:42:52 +05:30
dependabot[bot] 191027de7e
chore(deps)(deps): bump @chainsafe/libp2p-gossipsub (#1798)
Bumps [@chainsafe/libp2p-gossipsub](https://github.com/ChainSafe/js-libp2p-gossipsub) from 10.1.1 to 11.1.0.
- [Release notes](https://github.com/ChainSafe/js-libp2p-gossipsub/releases)
- [Changelog](https://github.com/ChainSafe/js-libp2p-gossipsub/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ChainSafe/js-libp2p-gossipsub/compare/v10.1.1...v11.1.0)

---
updated-dependencies:
- dependency-name: "@chainsafe/libp2p-gossipsub"
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-18 12:34:32 +01:00
Arseniy Klempner ae7bc3eeec
chore: bump all libp2p packages to latest version 2024-01-17 11:59:08 -08:00
Danish Arora b99f828cfb
chore: minor refactoring for protocols (#1762)
* remove: unnecessary function

* remove: test

* update doc

* add: tests

* revert unintended change

* fix: ping return

* fix(tests): stub connections

* log warning instead of throwing an error
2024-01-11 17:25:47 +05:30
fryorcraken aaa16f3ece
chore: release master (#1721) 2024-01-10 23:27:39 +01:00
Arseniy Klempner 69406bf1f0
reintroduce and deprecate named sharding (#1751)
Co-authored-by: danisharora099 <danisharora099@gmail.com>
2024-01-10 13:04:30 +05:30
danisharora099 2bc3735e4d
feat: add support for autosharded pubsub topics
tests: use a generator for sharded pubsub topics
set pubsub topic in encoder/decoder based on sharding type
add function for grouping content topics by pubsub topic
add autosharding config to create options
add autoshard rpc endpoints to nwaku and use in tests
set autoshard pubsub topics in all protocols
fix rebase with static sharding
removes unused function
remove console logs
remove autosharding from ShardInfo, add to EncoderOptions
fix enr and encoder/decoder options
test that same application/version hashes to same shard index
update comment on shard field
fix spelling of autosharding
fix content topic protocol in tests
add sharding type alias and function to determine topic in encoders/decoders
move DefaultPubsubTopic from core to interfaces
2023-12-21 10:03:22 -08:00
Danish Arora 9ac2a3f363
feat: metadata protocol (#1732)
* add proto

* add rpc and interfaces

* add protocol implementation

* update faulty proto def

* add rpc and interfaces

* refactor implementation & write test

* setup the metadata protocol as a service

* fix cases where metadata service needs to be undefined

* remove redundant catch block

* remove addressed TODO

* update import path

* log errors

* remove redundant code from handling incoming metadata request

* update tests

* add test to check for active connections

* change expects

* save remote peer's shard info after successful connection
2023-12-05 19:26:52 +05:30
dependabot[bot] e40e084429
chore(deps)(deps): bump @chainsafe/libp2p-gossipsub (#1744)
Bumps [@chainsafe/libp2p-gossipsub](https://github.com/ChainSafe/js-libp2p-gossipsub) from 10.1.0 to 10.1.1.
- [Release notes](https://github.com/ChainSafe/js-libp2p-gossipsub/releases)
- [Changelog](https://github.com/ChainSafe/js-libp2p-gossipsub/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ChainSafe/js-libp2p-gossipsub/compare/v10.1.0...v10.1.1)

---
updated-dependencies:
- dependency-name: "@chainsafe/libp2p-gossipsub"
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-30 00:38:21 +01:00
Danish Arora bbd372120a
chore: update all references of shard info to RFC terminology (#1740)
* update all references of shard info to RFC terminology

* bump nwaku to v0.22.0 (#1741)
2023-11-29 17:37:59 +05:30
Danish Arora 4cf2ffefa7
feat!: add support for sharded pubsub topics & remove support for named pubsub topics (#1697)
* merge branches

* tests: use a generator for sharded pubsub topics

* fix namespace edge case

* move shardInfo to pubsubTopic logic in waku.ts

* simplify encoder/decoder creation logic + update tests

* sharding utils: add error handling

* remove redundant test util

* baseprotocol: create abstraction for initialising pubsub topics

* fix: `createDecoder` interface

* filter: createSubscription takes shardInfo instead of pubsubTopicStr

* fix: sharding utils for error handling

* SingleShardInfo: use a new interface instead of omitting and rename namespace

* change redundant namespaces
2023-11-28 15:57:18 +05:30
Sasha 7df21b7756
chore: add info message with expected WebSocket connection failures (#1733)
* audit deps

* add info message

* add option to hide message

* add test

* remove only
2023-11-28 00:40:59 +01:00
Arseniy Klempner 1d0e2ace7f
feat: track node connection state (#1719)
Co-authored-by: chair <29414216+chair28980@users.noreply.github.com>
Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2023-11-27 12:44:49 +01:00