222 Commits

Author SHA1 Message Date
Arseniy Klempner
41301ea64a
feat: add v3 protocol support tests and exports
- Export LightPushCodecV3 and LightPushCodecs from core package
- Add v3 protocol support tests with status code validation
- Add mock functions for v3 responses and RLN errors
- Test mixed v2/v3 peer scenarios
- Validate protocol error mapping and status code handling
- Fix linting errors by adding explicit return types
2025-07-16 16:41:56 -07:00
Arseniy Klempner
b7bdecc760
feat: add light push v3 status codes and helpers 2025-07-16 16:41:53 -07:00
Sasha
ed389ccbc9
feat: add recovery and connection maintenance (#2496)
* add FF for auto recovery

* implement connection locking, connection maintenance, auto recovery, bootstrap connections maintenance and fix bootstrap peers dropping

* add ut for peer manager changes

* implement UT for Connection Limiter

* increase connection maintenance interval

* update e2e test
2025-07-17 01:15:36 +02:00
Sasha
35acdf8fa5
feat: don't dial peers that failed before, make dialer use dial queue (#2478)
* make dialer use dial queue

* skip undiable peers

* cover new cases in tests

* expose dialer config at connection manager

* update tests with new config

* add more tests
2025-07-15 23:13:58 +02:00
Sasha
27292edabc
feat!: unify events under one source (#2473)
* move health indicator under waku.events and expose from Waku as a value

* update tests

* make new type for libp2p event handlers

* fix types
2025-07-15 00:59:45 +02:00
Sasha
c7682ea67c
feat!: re-architect connection manager (#2445)
* remove public pubsub field and redundant util

* add hangUp and improve dial operations, improve keepAliveManager and remove unused method, move utils and add tests

* improve public dial method to start keep alive checks

* move dial method

* implement discovery dialer

* implement discovery dialer with queue with tests

* add discovery dialer e2e tests, change local discovery log tag, update other tests

* remove comment

* add issue link, remove only

* implement shard reader component

* create evetns module, remove unused connection manager events and related tests

* implement network indicator

* implement connection limiter, change public API of connection manager, implement recovery strategy

* decouple keep alive maanger

* add connection manager js-doc

* refactor keep alive manager, cover with tests

* add tests for connection manager main facade

* add tests for connection limiter

* add e2e tests for connection manager modules
pass js-waku config during test node init
remove dns discovery for js-waku

* restructure dialing tests

* address last e2e tests

* address review

* add logging for main methods

* decouple pure dialer class, update network monitor with specific metrics

* remove console.log

* remove usage of protocols

* update sdk package tests

* add connect await promise

* add debug for e2e tests

* enable only packages tests

* use only one file

* revert debugging

* up interface for netwrok manager

* add logs

* add more logs

* add more logs

* add another logs

* remove .only

* remove log statements

* skip the test with follow up
2025-07-09 21:23:14 +02:00
Sasha
981248eedd
chore!: remove deprecated filter implementation (#2433)
* chore: remove deprecated filter implementation

* update tests

* skip IReceiver test, remove unused utility

* fix typo

* address comments

* unskip Filter e2e test

* address more comments, remove duplication

* skip CI test
2025-07-02 12:37:54 +02:00
Sasha
cd1d909de3
chore: release master (#2390) 2025-06-23 22:29:40 +02:00
Sasha
f55db3eb4b
feat: shard retrieval for store and store peers selection (#2417)
* feat: implement shard retrieval for store and improve set store peers usage

* remove log

* remove only, improve condition

* implement smarter way to retrieve peers

* up tests

* update mock

* address nits, add target to eslint, revert to es2022
2025-06-23 10:01:54 +02:00
Sasha
fcc6496fef
chore!: upgrade libp2p, nodejs and typescript (#2401)
* remove react native package

* upgrade nodejs

* upgrade libp2p

* upgrade typescript, use 22 node

* up lock

* fix node compatibility

* upgrade playwright

* upgrade proto

* update tests
2025-06-22 17:41:23 +02:00
Sasha
7c8d1073b0
chore!: remove IBaseProtocol and improve interface on PeerExchange (#2422)
* remove IBaseProtocol

* fix references, interfaces and integration

* fix ci

* up mock

* up lock

* add mock for local storage

* add missing prop, fix tests

* up lock
2025-06-20 12:53:42 +02:00
Sasha
1905558753
feat!: re-work messaging parts and sharding (#2399)
* remove message-hash package, move it to core

* add js-doc to message hash

* up

* address type changes

* fix lint
2025-06-03 11:08:02 +02:00
Danish Arora
16328a3f11
feat: enable/disable discoveries without defaultBootstrap: true (#2363)
* feat: enable/disable discoveries without defaultBootstrap: true

* chore: rm .only

* chore: don't redefine default discoveries object

* chore: don't export default const from library

* chore: rename discoveriesEnabled to discovery

* fix: rename

* address comments

* chore: reset default

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>

* chore: address comments

* chore: update tests

---------

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2025-05-30 16:27:00 +05:30
Sasha
a4dfd3455c
feat: event based approach to Filter (#2300)
* create new filter api

* implement await on main methods on new Filter

* add info logs in new filter

* add logs to subscription impl

* remove lint supress

* add unit tests

* introduce E2E tests

* update e2e tests and add case for testing filter recovery after nwaku nodes replacement

* add new test cases for max limits and enable decoders as array on new filter

* fix edge case testing, correct test cases

* skip test

* update error message

* up text

* up text

* fix lint

* implement unsubscribeAll

* add js-dock to new filter

* add cspell

* implement TTL set for message history
2025-05-28 00:44:44 +02:00
Sasha
e22f211d46 chore: release master 2025-04-23 07:38:08 +02:00
Sasha
3038c48917
feat: introduce createDecoder and createEncoder on IWaku (#2352)
* feat: introduce createDecoder and createEncoder on IWaku

* add tests, refactor

* fix type
2025-04-14 10:46:47 +02:00
Sasha
ea6daae927
chore: release master (#2311) 2025-03-24 14:41:33 +01:00
Sasha
09108d9284
feat: enable auto start upon node creation (#2291)
* feat: enable auto start upon node creation

* update the state
2025-02-27 14:00:28 +01:00
Sasha
f199d92d60
feat: migrate to latest LightPush version (#2281)
* create retry manager

* update tests

* add retry manager tests, update peer manager

* fix start & merge with master

* return send to many logic

* add new error handling

* add sections to protocol errors

* fix check

* up test

* add waku.start in test

* fix check and test

* improve name
2025-02-25 22:40:03 +01:00
Sasha
3136f3a704
feat: add HealthIndicator with simplified logic and testing (#2251)
* implement HealthIndicator

* up libp2p interface version

* up lock

* remove unused tests

* expose HealthIndicator from Waku

* update test, add start and stop

* fix error handling
2025-02-05 13:24:50 +01:00
Sasha
62f93dc842
feat: improve peer manager and re-integrate to light push (#2191)
* up lock

* make ConnectionManager use ctor

* reform connection manager configurations

* remove log param from peerManager

* make PeerManager use only ConnectionManager, move getPeers to ConnectionManager, remove not needed code

* remove allPeers and connectedPeers from BaseProtocolCore, update tests, add getPeers for IWaku

* use only one peerManager from Waku object

* remove IBaseProtocolSDK and merge with PeerManager

* re-implement peerManager, remove ProtocolUseOptions

* remove not needed test, up lock

* update deps and lock

* remove old test for peerManager, fix check and spell

* rename to getConnectedPeers

* feat: improve filter subscriptions (#2193)

* add message cache to Filter

* remove WakuOptions and use only ProtocolCreateOptions

* move subscribe options to createLightNode Fitler protocol options

* rename SubscriptionManager to Subscription

* rename to CreateNodeOptions

* add warning

* feat: introduce subscription manager (#2202)

* feat: inroduce subscription manager

* fix: make pipeline succeed (#2238)

* fix test

* use hardcoded value

* update playwright

* fix test:browser

* up lock

* make peer retrieval probabilistic

* add comments

* up lightpush tests

* add tests for peer_manager, improve folder structure

* create named files for protocols

* create named files, simplify project structure

* remove only
2025-01-31 00:16:00 +01:00
Danish Arora
4153396014
feat(store): allow specifying node to use (#2192)
* feat: API allows using specific nodes for protocol

* chore: dial provided now

* chore: store uses the node

* chore: update API

* chore: use protocol-level class for getting peer

* chore: use ConnectionManager for dial ops

* chore: address comments

* chore: fix type error

* chore: Waku.dial() proxies through ConnectionManager.dialPeer

* chore: fix dial

* chore: add ts-doc for dialPeer()

* chore: remove log

* chore: reduce ts-doc for an internal function

* chore: address comments

* chore: return types from connmanager.dialpeer()

* chore: reduce diff by not introducing breaking changes
2025-01-28 17:57:49 +05:30
Danish Arora
7c0ce7b2ec
chore: upgrade to libp2p v2 (#2143)
* chore: upgrade libp2p to v2 and related deps

* chore: fix ENR

* chore(core): remove CustomEvent polyfill import

* chore: `peer-id-factory` has been removed

* chore(discovery): fix local-cache & remove CustomEvent imports

* chore(sdk): update config

* chore(tests): update tests without peer-id-factory

* fix: spec tests

* chore: fix test

* chore: upgrade dataset-core

* chore: upgrade libp2p and stale references

* chore: upgrade playwright

* chore: rm console log

* fix: lock
2024-10-21 16:43:24 +05:30
Sasha
c591e6d294
chore: release master (#2135) 2024-10-17 14:06:38 +02:00
Sasha
ded994f8ec
fix: attempt to fix some of the Filter issues (#2183)
* feat: lighten retry logic for LightPush

* update tests

* remove base protocol sdk from light push, add unit tests for light push

* remove replaced test

* ensure numPeersToUse is respected

* turn off check for missing messages

* fix recurring ping

* add useful logs

* skip tests

* remove comment

* feat: check filter subscriptions against lightPush (#2185)
2024-10-17 01:01:21 +02:00
Sasha
4049123f14
feat!: lighten retry logic for LightPush (#2182)
* feat: lighten retry logic for LightPush

* update tests

* remove base protocol sdk from light push, add unit tests for light push

* remove replaced test

* ensure numPeersToUse is respected

* skip tests
2024-10-17 00:49:24 +02:00
Danish Arora
b2efce5ec2
feat(filter): enhancing protocol peer management with mutex locks (#2137)
* chore: improvements

* chore: add logs for subscription maintenance

* chore: update logging

* chore: trimming down BaseProtocolCore

* chore: track peers in a hashmap instead of array

* chore: peer mgmt responds to conenction/disconnection and improve logging

* feat: add mutex locks to tackle race conditions over shared state

* fix: build

* chore: some mutex lock-release improvements

* feat: peer manager

* chore: rm tests for remove internal util

* chore: update HealthManager updates

* chore: update tests

* rm: only

* fix: hasPeers management

* chore: add modularity to getting connected peers

* chore: improve logs & add debug

* chore: renewal doesnt disconnect, only removes

* chore: await for sequential operations

* chore: add TODO

* chore: minor improvements

* chore: fix rebase

* chore: update playright

* chore: remove additional arg

* chore: update interafce

* feat(peer-manager): unit tests

* chore: improve hasPeers()

* chore: update lockfile

* feat: Filter reacts to peer:disconnect event, add tests

* chore: fix lock

* chore: update playright

* chore: update protocol health for lightpush

* chore: remove .only

* chore: address comments and improvements

* fix: tsconfig
2024-10-10 23:47:12 +02:00
Sasha
75fcca4cd9
feat: replace waitForRemotePeers() with waku.waitForPeer() method (#2161)
* fix comment of default number of peers

* export default number of peers from base protocol sdk

* rename to light_push, move class to separate file

* move waitForRemotePeer to sdk package

* add todo to move waitForGossipSubPeerInMesh into @waku/relay

* clean up waitForRemotePeer, split metadata await from event and optimise, decouple from protocol implementations

* simplify and rename ILightPush interface

* use only connected peers in light push based on connections instead of peer renewal mechanism

* improve readability of result processing in light push

* fix check & update tests

* address tests, add new test cases, fix racing condition in StreamManager

* use libp2p.getPeers

* feat: confirm metadata and protocols needed in waitForRemotePeer

* rely on passed protocols and fallback to mounted

* add I prefix to Waku interface

* implement waku.connect method

* add docs to IWaku interface

* remove export and usage of waitForRemotePeer

* move wait for remote peer related to Realy out of @waku/sdk

* change tests to use new API

* fix linting

* update size limit

* rename .connect to .waitForPeer

* export waitForRemotePeer and mark as deprecated

* feat: add mocha tests to @waku/sdk and cover waitForRemotePeer (#2163)

* feat: add mocha tests to @waku/sdk and cover waitForRemotePeer

* add waitForRemote UTs

* remove junk

* feat: expose peerId and protocols from WakuNode (#2166)

* chore: expose peerId and protocols from WakuNode

* remove unused method

* move to private method

* rename to waitForPeers

* up test
2024-10-09 00:43:34 +02:00
Sasha
2be0e81a0a
chore: rename IProtocolSDK interfaces to IProtocol naming convention (#2159)
* chore: rename IProtocolSDK interfaces to IProtocol
2024-10-04 13:53:54 +02:00
Sasha
1d68526e72
feat(lightPush): improve peer usage and improve readability (#2155)
* fix comment of default number of peers

* export default number of peers from base protocol sdk

* rename to light_push, move class to separate file

* move waitForRemotePeer to sdk package

* add todo to move waitForGossipSubPeerInMesh into @waku/relay

* clean up waitForRemotePeer, split metadata await from event and optimise, decouple from protocol implementations

* simplify and rename ILightPush interface

* use only connected peers in light push based on connections instead of peer renewal mechanism

* improve readability of result processing in light push

* fix check & update tests

* address tests, add new test cases, fix racing condition in StreamManager

* use libp2p.getPeers
2024-10-04 10:50:58 +02:00
Danish Arora
7ad1d321ca
feat(filter): reliability monitor as a separate class to handle reliability logic (#2117)
* chore: move SubscriptionManager to a separate file

* feat: introduce ReliabilityMonitor

* fix: peer data updates

* fix: logical error when returning includesMessage

* chore: handle edge case

* chore: move ping failures handling inside monitor

* chore: move renewal logic to monitor

* chore: improve structuring

* chore: update logger

* chore: readd connectionListener() logic from merge

* chore: minor fixes

* chore: improve

* chore: setup destruction of ReliabilityMonitors

* fix: condition for ping failure

* fix: setters

* chore: handle race condition & fix test
2024-09-13 14:57:29 +05:30
Danish Arora
9d9a696024
chore: mandate .js extensions for imports (#2128)
* feat: add eslint rule

* chore: update codebase import extensions
2024-09-13 14:18:29 +05:30
Sasha
a2c5dba32d
chore: release master (#2122) 2024-09-06 01:42:16 +02:00
Sasha
0263cb80c5
fix: improve node bootstrapping (#2121)
* fix: improve node bootstrapping

* implement dns discovery on failed dials

* update const to match RFC

* up
2024-09-06 01:10:54 +02:00
Sasha
5cfe9327b6
chore: remove relay from sdk package (#2040)
* remove full node

* remove relay from sdk

* comment node counter check

* try using logline

* up comment
2024-08-31 15:18:51 +02:00
Sasha
f9361bf774
chore: release master (#2066) 2024-08-29 11:49:04 +02:00
Sasha
a022433851
feat(lightpush)!: return new error messages (#2115)
* feat: return new error messages

* fix test, remove unused

* up

* up

* rollback

* up test
2024-08-29 11:20:19 +02:00
Danish Arora
69b6612ec9
chore: upgrade node to LTS 20 (#2110)
* chore: upgrade nodejs to new LTS 20

* chore: update other references
2024-08-23 15:57:20 +05:30
Sasha
2beb359cef
chore: enabile back filterAll if set by consumer (#2103) 2024-08-22 15:55:29 +02:00
Danish Arora
409642d7d3
chore: upgrade libp2p and related packages (#2084)
* chore: upgrade libp2p and related packages

* fix: public key on peer id can now be undefined

* chore: upgrade remainder packages

* chore: fix tests

* chore: address comments

* chore: upgrade playwright and proto
2024-08-14 04:23:04 +05:30
Danish Arora
5ce36c8f18
feat!: deprecate named pubsub topics and use static/auto sharding (#2097)
* feat: deprecate named sharding & protocols adhere
simplify network config type, all protocols use pubsub topic internally

* chore: update tests

* tests: rm application info

* chore: use static sharding and auto sharding terminologies

* chore: update docs for network config

* chore: update interfaces

* tests: update tests error message

* chore: remove `ShardingParams` type and fix test
2024-08-13 05:23:20 +05:30
Danish Arora
86f730f958
feat!: store v3 (#2036)
* feat: write proto

* chore: move store v2 to a subdir

* chore: update v3 proto

* feat: create custom RPC

* feat: implement storev3 core

* chore: set store v3 as default

* chore: move v2 related code

* chore: update v2 imports

* feat: add store-v3 sdk implementation

* fix: rebase

* chore: add ts-doc for store query request params

* chore: update tests for new API

* fix: use nanoseconds instead of millisecond for timerange

* chore: improve store

* chore: remove store v2

* chore: update tests

* chore: fix legacy imports & proto

* tests: remove manual reversal as its part of the API, update incorrect cursor error msg

* chore: update default page size

* chore: account for MAX_PAGE_SIZE from nwaku

* fix: test

* fix: sorting tests
2024-08-06 12:06:37 +05:30
Danish Arora
fdd9dc44a4
feat(filter)!: new simpler filter API (#2092)
* chore: rename IReceiver subscribe

* feat!: new `subscribe() API that only takes in decoders and callback

* chore: `to_async_iterator` uses new function name

* chore: make `createSubscription` private, and shorten error handling

* chore: update subscribe return type

* tests: use new API

* fix: tests
2024-08-05 15:52:58 +05:30
Danish Arora
d464af3645
feat: node and protocols health (#2080)
* feat: introduce HealthManager

* feat: make health accessible on Waku object

* feat: update health from protocols

* chore: add access modifiers to healthmanager

* feat: use a HealthManager singleton

* chore: add tests for Filter, LightPush and Store

* feat: add overall node health

* chore: update protocol health to consider Store protocol

* chore: setup generic test utils instead of using filter utils

* tests: add a health status matrix check from 0-3

* chore: increase timeout for failing tests in CI
tests pass locally without an increased timeout, but fail in CI

* chore: move name inference to HealthManager

* tests: abstract away node creation and teardown utils

* fix: import
2024-07-27 18:27:54 +05:30
gabrielmer
08fc2d133a
chore: changing default pubsub topic to its static sharding version (#2083)
* changing default pubsub topic to its static sharding version

* keeping RFC's Waku Message test vectors

* reverting change in changelog

* setting pubsub topic when creating nwaku node

* adding shardInfo to runMultipleNodes call

* adding shardInfo to runMultipleNodes call in lightpush tests

* add pubsub topics to nwaku.start

* get rid of it.only that remained

* fixing compliance tests

* setting clusterId to 0

* removing unnecessary fix

* adding shardInfo when creating nodes

* fixing wait for remote peer tests

* fixing peer exchange test

* refactor

* removing unnecessary variable

* feat: create default shard info, update tests (#2085)

* feat: create default shard info, update tests

* add link

* fix tests

* remoe only

* up tests

* up test

---------

Co-authored-by: Sasha <118575614+weboko@users.noreply.github.com>
2024-07-26 00:21:52 +02:00
Danish Arora
9b0f1e855a
feat: validate messages for individual filter nodes & perform renewals (#2057)
* feat: validate messages for individual filter nodes & perform renewals

* chore: fix spell check

* chore: use a max threshold before peer renewal

* chore: switch from a validation cycle timer to adhoc validation

* chore: add test

* fix: test

* chore: address comments

* fix: renewal without a new peer available

* chore: validating messages should be non-blocking

* chore: minor improvements

* chore: rm only

* chore: fix test
2024-07-18 00:41:21 +05:30
Sasha
ccdf84e593
chore: release master (#2004)
reliability improvements
2024-07-10 17:25:22 +02:00
Danish Arora
318667e442
feat(filter): peer/subscription renewal with recurring Filter pings (#2052)
* chore: renewPeer() returns the new found peer

* feat: ping & peer renewal

* chore: add tests

* fix: tests

* chore: remove only

* chore: remove comments

* chore(tests): decrease timeout

* chore: add array index validation

* chore: remove only

* chore: move defaults into a separate variable

* chore: update lightpush with new API

* chore: include peer renewals within `ping` instead of `interval`

* chore: update tests

* chore: add new test

* chore: address comments
2024-07-10 15:34:16 +05:30
Danish Arora
4db508b962
feat(filter): use protocol peer management (#2047)
* feat: leverage protocol peer management

* chore: add test

* chore: address comments

* chore: add todo
2024-07-03 12:09:34 +05:30
Danish Arora
93e78c3b87
feat(lightpush): peer management for protocols (#2003)
* chore: make `dropConnection` to be a public function

* feat: peers are maintained for protocols
- passes `ConnectionManager` to ProtocolSDK to allow disconnecting from within protocol
- maintains `numPeersToUse` for each protocol within BaseProtocolSDK

* fix: pass options to protocols

* chore: update interfaces to allow public access

* chore: improve logging on protocol

* fix: renew peer upon failure

* chore(tests): allow DefaultPubsubTopic

* feat(lightpush): write peer management tests

* chore: rename test

* feat: add lock to `maintainPeers()` to handle parallelisation of requests
fixes parallelisation of lightpush.send() requests

* fix: concurrent lightpush requests

* fix: test & improve peers fetching

* chore: use getter

* address comments

* chore: smaller improvements

* feat: attempt to improve time for first lightpush.send()

* chore: use `window.interval` for type-safety

* chore: remove delays

* feat: add autoRetry

* feat: `forceUseAllPeers` to wait for all connected peers to be resoled
2024-06-19 01:52:16 -04:00