44 Commits

Author SHA1 Message Date
Danish Arora
73d4f19746
fix(message-hash): account for timestamp (#1986)
* fix: message-hash algo + tests

* rm: only

* chore: move numberToBytes to utils package

* chore: shorten implementation
2024-04-29 15:35:23 +05:30
Sasha
f3627c46a4
feat!: use ShardingParams on subscriptions, make Decoder/Encoder auto sharding friendly by default (#1958)
* fix: use pubsubTopic from current ones if not set

* fix: improve type on dial method

* enforce same pubusb on filter.subscribe, make content topic to pubsub mapping default for decoder / encoder

* fix mapping problem

* update tests

* add error handling

* fix typo

* up lock

* rm lock

* up lock

* remove only

* fix content topic

* fix ephemeral test

* fix filter unsubscribe test

* up utils

* fix subscribe test

* up interfaces and filter api

* remove only

* up ping test

* fix subscribe test

* fix push test

* fix lightPush

* fix multiple pubsub

* remove only, fix subscribe filter test

* remove only

* fix cluster ID selection and named sharding subscription test

* fix unsubscribe test

* fix light push test

* fix light push test

* fix push test

* fix relay publish

* create runNode and fix relay tests

* generalize runNodes, fix some tests

* fix store tests

* fix toAsyncIterator tests

* remove only

* fix lightPush

* use generics

* try fix test

* run failing tests

* remove only

* address failed tests, remove DefaultPubsubTopic dependency in some tests
2024-04-28 11:15:17 +02:00
Arseniy Klempner
9514653428
fix: use correct shard index when creating encoder 2024-04-24 18:40:17 -07:00
Florin Barbu
f4c7c02f83
test: sharding tests refactor (#1883)
* sharding tests refactor

* small fixes

* adjust clusterID based on version

* fix typo

* fix dispatchEvent test

* sharding unit tests

* port adjustment

* update unit tests

* fix 1902

* adjust content topic tests

* adjust metdata tests

* small adjustments

* fix

* update resolveAutoshardingCluster version

* skip autosharding tests for nwaku < 0.27.0

* skip autosharding tests for nwaku < 0.27.0
2024-03-18 13:40:08 +02:00
Danish Arora
1882023c58
feat(metadata): use error codes (#1904) 2024-03-12 16:40:08 +05:30
Danish Arora
8deab11890
chore(lightpush)!: move protocol implementation opinions to @waku/sdk (#1887)
* chore: restructure @waku/sdk

* chore: introduce `BaseProtocolCore` and `BaseProtocolSDK`

* chore: introduce `LightPushCore` and `LightPushSDK`

* chore: update `relay` for new types

* chore(sdk): update structure

* chore(filter): add `numPeersToUse`

* chore: update tests

* update: size-limit

* chore: update more tests

* attach issue link to TODOs
2024-03-11 18:50:34 +05:30
Danish Arora
199f6ab2ff
feat: local discovery (#1811)
* initialise the new package

* feat: implement LocalStorageDiscovery

* add: Startable to cspell

* add compliance test

* add: discovery tests

* rm: browser tests script

* address comments

* add type safety to getting peers from local

* only dispatch peer if it does not exist

* move ws ma extraction to utils

* chore: update package name to local-discovery

* fix: add compliance test with no external deps on service node

* use peer:identify instead of peer:update

* add: unit tests & remove sdk dependency

* move tests to self package

* update cspell + remove unrequired deps

* add types

* maintain in-memory peers for localstorage

* address comments

* chore: rename

* use name from options

* fix: saving peers

* rm: only
2024-02-16 20:06:27 +05:30
Arseniy Klempner
7e88d3b565
chore: add autosharding test vectors 2024-02-08 10:42:41 -08:00
Arseniy Klempner
e138b4f5c4
feat: decouple sharding params out of core 2024-02-08 08:42:47 -08:00
Arseniy Klempner
68d3229644
feat: make ShardingParams optional in sdk, required internally 2024-01-24 07:36:00 -08:00
Danish Arora
477c2a5918
feat!: protocols filter peers as per configured shard (#1756)
* merge: master

* fix: tests

* update: interfafces

* rm: comments

* metadata: store peerIdStr instead of peerId

* chore(utils): move fast-utils to dev deps

* fix: allow autosharding nodes to get peers (#1785)

* fix: merge

* fix: build

* fix: failing tests from master merge

---------

Co-authored-by: Arseniy Klempner <arseniyk@status.im>
2024-01-19 20:42:52 +05:30
Arseniy Klempner
ae7bc3eeec
chore: bump all libp2p packages to latest version 2024-01-17 11:59:08 -08:00
Danish Arora
b99f828cfb
chore: minor refactoring for protocols (#1762)
* remove: unnecessary function

* remove: test

* update doc

* add: tests

* revert unintended change

* fix: ping return

* fix(tests): stub connections

* log warning instead of throwing an error
2024-01-11 17:25:47 +05:30
Arseniy Klempner
69406bf1f0
reintroduce and deprecate named sharding (#1751)
Co-authored-by: danisharora099 <danisharora099@gmail.com>
2024-01-10 13:04:30 +05:30
Danish Arora
9983549e8b
chore: use connected peers instead of discovered peers for protocols (#1758)
* switch to use connected peers instead of discovered peers for protocols

* switch to use connected peers instead of discovered peers for protocols
2024-01-02 15:49:31 +05:30
danisharora099
2bc3735e4d
feat: add support for autosharded pubsub topics
tests: use a generator for sharded pubsub topics
set pubsub topic in encoder/decoder based on sharding type
add function for grouping content topics by pubsub topic
add autosharding config to create options
add autoshard rpc endpoints to nwaku and use in tests
set autoshard pubsub topics in all protocols
fix rebase with static sharding
removes unused function
remove console logs
remove autosharding from ShardInfo, add to EncoderOptions
fix enr and encoder/decoder options
test that same application/version hashes to same shard index
update comment on shard field
fix spelling of autosharding
fix content topic protocol in tests
add sharding type alias and function to determine topic in encoders/decoders
move DefaultPubsubTopic from core to interfaces
2023-12-21 10:03:22 -08:00
Danish Arora
bbd372120a
chore: update all references of shard info to RFC terminology (#1740)
* update all references of shard info to RFC terminology

* bump nwaku to v0.22.0 (#1741)
2023-11-29 17:37:59 +05:30
Danish Arora
4cf2ffefa7
feat!: add support for sharded pubsub topics & remove support for named pubsub topics (#1697)
* merge branches

* tests: use a generator for sharded pubsub topics

* fix namespace edge case

* move shardInfo to pubsubTopic logic in waku.ts

* simplify encoder/decoder creation logic + update tests

* sharding utils: add error handling

* remove redundant test util

* baseprotocol: create abstraction for initialising pubsub topics

* fix: `createDecoder` interface

* filter: createSubscription takes shardInfo instead of pubsubTopicStr

* fix: sharding utils for error handling

* SingleShardInfo: use a new interface instead of omitting and rename namespace

* change redundant namespaces
2023-11-28 15:57:18 +05:30
Arseniy Klempner
86da6962ba
feat: add function for determining shard index from content topic 2023-11-16 00:31:50 -08:00
Arseniy Klempner
1bc1eb5091
feat: add function to validate autoshard content topic 2023-11-15 22:28:25 -08:00
Danish Arora
3166a5135e
chore!: change all instances of PubSubTopic to PubsubTopic (#1703)
* rename all PubSub patterns

* feat: forbid identifiers with camelcase pubSub (#1709)

---------

Co-authored-by: Arseniy Klempner <adklempner@gmail.com>
2023-11-14 18:52:52 +03:00
Danish Arora
49a3666208
revert logger namespace (#1696) 2023-10-30 19:16:03 +05:30
fryorcraken
b7dc3d7576
fix: measure total message size (#1643)
Network message limitations are imposed on the whole message, not just
the payload.

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2023-10-26 13:14:47 +02:00
Danish Arora
81b2bf46ab
allow filtering based on log levels (#1677) 2023-10-23 13:26:35 +05:30
Danish Arora
0f7d63ef93
feat: Logger with log levels (#1672)
* setup a custom Logger with log level support

* refactor codebase for to use new Logger with log levels

* disallow usage of `debug` directly / only allow usage in/through custom Logger

* remove `debug` from logger
2023-10-20 16:36:47 +05:30
fryorcraken
6bad4ea7d1
feat: fail early when trying to send empty payload (#1642)
* refactor: rename as this method does not check empty payloads

* feat: fail early when trying to send empty payload

---------

Co-authored-by: Sasha <oleksandr@status.im>
2023-10-17 00:14:45 +02:00
Danish Arora
124a29ebba
feat(static-sharding): filter peer connections per shards (#1626)
* add interface for `ShardInfo`

* enr: add deserialization logic & setup getters

* add sharding related utils

* utils: add shard<-> bytes conversion helpers

* pass `pubSubTopics` to `Waku`

* add `rs`/`rsv` details during discovery

* connection-manager: discard irrelevant peers

* add tests for static sharding - peer exchange

* update `ConnectionManager` tests to account for topic validity

* add js suffix to import

* address some comments

* move shardInfo encoding to ENR

* test: update for new API

* enr: add tests for serialisation & deserialisation

* address comment

* update test

* move getPeershardInfo to ConnectionManager and return ShardInfo instead of bytes

* update encoding and decoding relay shards to also factor for shards>64

* relay shard encoding decoding: use DataView and verbose spec tests

* improve tests for relay shard encoding decoding

* rm: only

* improve log message for unconfigured pubsub topic

* minor improvement

* fix: buffer <> Uint8array problems with shard decoding

* fix: test

* rm: only
2023-10-10 20:18:02 +05:30
Danish Arora
a3c45b6e1a
feat(static-sharding)!: allow multiple pubSubTopics (#1586)
* `ProtocolCreateOptions` now has `pubSubTopic` as `pubSubTopic[]`

* chore: update encoder & decoder to support `PubSubTopic`

* feat(protocols): allow multiple `PubSubTopic[]`

* feat(relay): allow multiple `PubSubTopic[]`

* chore(tests): update for new API

* chore: minor fixes

* chore: make store more robust

* fix(relay): correctly set types

* chore(address comments): update terminology around configured pubsub topics

* chore(address comments): minor refactoring

* chore(relay): split `subscribe` into smaller functions for readability & modularity

* chore(address comments): refactor `waitForGossipSubPeerInMesh`

* chore(store): only allow to query one `pubSubTopic`

* fix: `store` bug

* feat(tests): add some basic tests

* sharding utils

* address comments

* feat(relay): re-add API for `getMeshPeers`

* update error message

Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>

* refactor for new API

* feat: simplify handling of observers (#1614)

* refactor: simplify handling of observers

* refactor: Remove redundant PubSubTopic from Observer

* use `??` instead of `||`

* update `pubsubTopic` to `pubSubTopic`

* update `interval` typo

* change occurence of `pubsubTopic` to `pubSubTopic`

* relay: rm `getAllMeshPeers` and make `pubSubTopics` public

* relay: use `push_or_init_map` and move to `utils`

* fix: update API for tests

* fix: relay waitForRemotePeer

---------

Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>
2023-09-27 15:28:07 +05:30
Danish Arora
6f09fbf4ed
feat: use the lowest latency peer for protocols (#1540)
* maintain pings in a hashmap

* convert `KeepAliveManager` into a singleton

* chore: fix an unrelated cyclic dependency error

* update `selectPeerForProtocol` to return peer with the lowest latency

* use the new KeepAliveManager API

* use the new API for `selectPeerForProtocol`

* add tests

* use PeerData to hold the ping instead of a new variable

* improve tests for readability

* move back KeepAliveManager from singleton

* reenable all tests

* minor improvements

* improve error handling

* convert .then() syntax to async/await
2023-09-08 21:36:55 +05:30
Danish Arora
81a52a8097
feat!: set peer-exchange with default bootstrap (#1469)
* set peer-exchange with default bootstrap

* only initialise protocols with bootstrap peers

* update package

* update package-lock

* refactor `getPeers` while setting up a protocol

* move codecs to `@waku/interfaces`

* lightpush: send messages to multiple peers

* only use multiple peers for LP and Filter

* fix: ts warnings

* lightpush: tests pass

* update breaking changes for new API

* move codecs back into protocol files

* refactor: `getPeers()`

* rm: log as an arg

* add tsdoc for getPeers

* add import

* add prettier rule to eslint

* add: peer exchange to sdk as a dep

* fix eslint error

* add try catch

* revert unecessary diff

* revert unecessary diff

* fix imports

* convert relaycodecs to array

* remove: peerId as an arg for protocol methods

* keep peerId as an arg for peer-exchange

* remove: peerId from getPeers()

* lightpush: extract hardcoded numPeers as a constant

* return all peers if numPeers is 0 and increase readability for random peers

* refactor considering more than 1 bootstrap peers can exist

* use `getPeers`

* change arg for `getPeers` to object

* address comments

* refactor tests for new API

* lightpush: make constant the class variable

* use `maxBootstrapPeers` instead of `includeBootstrap`

* refactor protocols for new API

* add tests for `getPeers`

* skip getPeers test

* rm: only from test

* move tests to `base_protocol.spec.ts`

* break down `getPeers` into a `filter` method

* return all bootstrap peers if arg is 0

* refactor test without stubbing

* address comments

* update test title

* move `filterPeers` to a separate file

* address comments & add more test

* make test title more verbose

* address comments

* remove ProtocolOptions

* chore: refactor tests for new API

* add defaults for getPeers

* address comments

* rm unneeded comment

* address comment: add diversity of node tags to test

* address comments

* fix: imports
2023-09-07 13:15:49 +05:30
Danish Arora
87717981eb
chore: upgrade libp2p and related deps (#1482)
* chore: update noise

* update: package.lock

* update: @chainsafe/libp2p-gossipsub

* rm unwanted libp2p interface deps & bump up libp2p

* refactor code for new deps

* update: new package.lock

* setup prettier, refactor eslint  and rm trailing commas

* update package.lock

* fix build

* import type for interface

* fix imports for merge

* update typedoc exports

* add: CustomEvent import

* use new libp2p interface

* add aegir as dev dep for tests
2023-08-16 20:18:13 +05:30
Danish Arora
7b6ead14ac
update: tsdoc (#1476) 2023-08-16 16:11:27 +05:30
Sasha
51f9261a16
chore: update dependencies (#1460)
* update to latest prettier

* fix with prettier: added trailing comma

* remove deps from test package, add sinon types, fix type hack in a test, update esling prettier config

* update typescript eslint plugins

* update package-locks
2023-08-11 15:14:02 +02:00
Danish Arora
8d0e647966
feat!: filter v2 (#1332)
* implement proto

* implement filter v2

* add tests

* minor improvements
- make unsubscribe functions private in
filter
- enable all tests

* enable all tests

* readd multiaddrinput

* address comment removals

* unsubscribe based on contentFilters passed

* update unsubscribe function parameters in test

* reset interfaces & filter v1

* refactor filterv2 into 2 classes
- removes generics from types on filter which means
manual typecasting to filter version is required on
consumer side
- defaults to filterv2
- splits filterv2 into 2 classes:
  - one to create the subscription object with a
peer which returns the second class
  - the other to manage all subscription functions

* updates filter tests for the new API
- also fixes an interface import

* update `toAsyncIterator` test for Filter V1

* implement IReceiver on FilterV2

* remove return values from subscription functions

* update `to_async_iterator`

* address variable naming

* add tsdoc comments for hidden function

* address minor comments

* update docs to default to filter v2

* address comments

* rename `wakuFilter` to `wakuFilterV1`

* chore: Remove static variables (#1371)

* chore: Remove static variables

- Remove internal types from `@core/interfaces`
- Remove data being redundantly stored (pubsub topic)
- Remove usage of static variables
- Clean up callbacks and decoders when using `unsubscribe`
- Clean up callbacks and decoders when using `unsubscribeAll`

* fix setting activeSubscription

---------

Co-authored-by: danisharora099 <danisharora099@gmail.com>

* make activeSub getter and setter private

* update size-limit

---------

Co-authored-by: fryorcraken.eth <110212804+fryorcraken@users.noreply.github.com>
2023-05-23 16:06:46 +05:30
Sasha
72f97d4545
feat: add 1MB restriction to LightPush and Relay (#1351)
* add 1MB restriction to LightPush and Relay

* fix condition

* improve lightPush test

* update test

* add isomorphic-webcrypto

* import module

* add errors to SendResult and tests

* fix lint
2023-05-17 23:40:52 +02:00
Sasha
ef87af6e7a
chore: add iterator timeout and unblock main thread (#1357)
* add iterator timeout
* add idle state to generator
* add js-doc and iterator option
2023-05-17 10:38:20 +02:00
Danish Arora
50c2c2540f
feat!: @waku/relay (#1316)
* move constants & core's func to @waku/utils

* setup @waku/relay

* use @waku/relay for create

* address: principal review

* fix: size-limit

* move all constants to @waku/core

* move TopicOnlyDecoder to relay

* merge: master

* fix: tests

* Move constants out of core (#1340)

* update package.lock

* change constants folder into a ts file

* use dependency version as * instead of version number

* add constants import from @waku/core

* add empty changelog file for relay

---------

Co-authored-by: fryorcraken.eth <110212804+fryorcraken@users.noreply.github.com>
2023-05-11 14:08:00 +05:30
Sasha
7daa9d05bf
feat: toSubscriptionIterator impl for IReceiver (#1307) 2023-05-09 20:15:37 +02:00
fryorcraken.eth
3a92355543
chore: code improvement
- Replace exception pattern with a result pattern as it better fits the
usage.
- merge few maps to prefer chaining to assign var after var.
- Make `isDefined` type helper a common util function
2023-03-21 08:59:25 +11:00
Danish Arora
a30b2bd747
chore!: add exports map to @waku/utils (#1201)
* add exports map

* merge: master

* export  subdir with default export path

* import according to the export path
2023-03-14 10:10:38 +05:30
fryorcraken.eth
6dbcde041a
chore!: directly convert from ENR to PeerInfo, remove unneeded utility 2023-03-10 10:23:39 +11:00
fryorcraken.eth
7d29ed1d99
chore: move error throwing within selectPeerForProtocol
As all callers throw upon undefined result.
2023-02-27 09:48:30 +11:00
Danish Arora
6b87ca1c4d
chore: move random_subset.ts to @waku/utils (#1191) 2023-02-26 12:31:55 -07:00
Danish Arora
b33306655a
chore: move libp2p and byte utils to a single utils package (#1149)
* move libp2p and byte utils to a single `utils`
package

* set package version to 0.0.1
2023-02-06 11:22:36 +05:30