* feat(sds): messages with lost deps are delivered
This is to re-enable participation in the SDS protocol. Meaning the
received message with missing dependencies becomes part of the causal
history, re-enabling acknowledgements.
* fix(sds): avoid overflow in message history storage
* feat(reliable-channel): Emit a "Synced" Status with message counts
Return a "synced" or "syncing" status on `ReliableChannel.status` that
let the developer know whether messages are missing, and if so, how many.
* fix: clean up subscriptions, intervals and timeouts when stopping
# Conflicts:
# packages/sdk/src/reliable_channel/reliable_channel.ts
* chore: extract random timeout
* fix rebase
* revert listener changes
* typo
* Ensuring no inconsistency on missing message
* test: streamline, stop channels
* clear sync status sets when stopping channel
* prevent sync status event spam
* test: improve naming
* try/catch for callback
* encapsulate/simplify reliable channel API
* sanity checks
* test: ensure sync status cleanup
* chore: npm publication
Fixing npm publication and warnings
* Upgrade workflow to use trusted publishing
https://docs.npmjs.com/trusted-publishers
* bump node js to 24
To avoid having to reinstall npm in pre-release for npmjs trusted publishers
* SDS: pushOutgoingMessage is actually sync
* SDS: ensure that `ContentMessage` class is stored in local history with `valueOf` method
* feat: introduce reliable channels
Easy to use Scalable Data Sync (SDS, e2e reliability) wrapper, that includes:
- store queries upon connection to store nodes
- store queries to retrieve missing messages
* remove `channel` prefix
* attempt to improve performance when processing a lot of incoming messages
* test: split test file
* use index.ts for re-export only.
* improve if condition
* use getter for isStarted
* waku node already auto-start
* rename send
* fix lightPush.send type post rebase
* test: remove extra console.log
* SDS: emit messages as missing as soon as they are received
* make configurable elapse time for task process
* typo
* use string instead of enum for event types
* ReliableChannel.send returns the message id
* update local peer discovery, make it configurable for cache
* move to separate file
* up tests, remove local storage from tests
* pass local peer cache options
* add e2e tests
* add aditional e2e tests for local cache
* rename local-peer-cache into peer-cache
* update tests, ci
* prevent filterign ws addresses
These are high level configuration values, that need to be consistent within a given execution. Setting them as default value at such a low level is a footgun risk.
Concepts are being mixed up between the global network config (static vs auto sharding), that needs to be the same of all nodes in the network, individual node configuration (eg relay node subscribing to a given shard), and the routing characteristic of a specific message (eg pubsub topic, shard).
This stops proper configuration of nwaku post 0.36.0 because we know need to be deliberate on whether nwaku nodes are running with auto or static sharding.
It also included various back and forth conversions between shards, pubsub topics, etc.
With this change, we tidy up the network configuration, and make it explicit whether it is static or auto sharded.
We also introduce the concept of routing info, which is specific to a message, and tied to the overall network configuration.
Routing info abstract pubsub topic, shard, and autosharding needs. Which should lead to easier tidy up of the pubsub concept at a later stage.
# Conflicts:
# packages/core/src/lib/connection_manager/connection_manager.ts
# packages/core/src/lib/metadata/metadata.ts
# packages/interfaces/src/metadata.ts
# packages/interfaces/src/sharding.ts
# packages/relay/src/create.ts
# packages/sdk/src/filter/filter.ts
# packages/sdk/src/filter/types.ts
# packages/sdk/src/light_push/light_push.spec.ts
# packages/tests/tests/sharding/auto_sharding.spec.ts
# packages/tests/tests/sharding/static_sharding.spec.ts
# Conflicts:
# packages/sdk/src/store/store.ts
* up lock
* make ConnectionManager use ctor
* reform connection manager configurations
* remove log param from peerManager
* make PeerManager use only ConnectionManager, move getPeers to ConnectionManager, remove not needed code
* remove allPeers and connectedPeers from BaseProtocolCore, update tests, add getPeers for IWaku
* use only one peerManager from Waku object
* remove IBaseProtocolSDK and merge with PeerManager
* re-implement peerManager, remove ProtocolUseOptions
* remove not needed test, up lock
* update deps and lock
* remove old test for peerManager, fix check and spell
* rename to getConnectedPeers
* feat: improve filter subscriptions (#2193)
* add message cache to Filter
* remove WakuOptions and use only ProtocolCreateOptions
* move subscribe options to createLightNode Fitler protocol options
* rename SubscriptionManager to Subscription
* rename to CreateNodeOptions
* add warning
* feat: introduce subscription manager (#2202)
* feat: inroduce subscription manager
* fix: make pipeline succeed (#2238)
* fix test
* use hardcoded value
* update playwright
* fix test:browser
* up lock
* make peer retrieval probabilistic
* add comments
* up lightpush tests
* add tests for peer_manager, improve folder structure
* create named files for protocols
* create named files, simplify project structure
* remove only
* chore: rename IReceiver subscribe
* feat!: new `subscribe() API that only takes in decoders and callback
* chore: `to_async_iterator` uses new function name
* chore: make `createSubscription` private, and shorten error handling
* chore: update subscribe return type
* tests: use new API
* fix: tests
* chore: make `dropConnection` to be a public function
* feat: peers are maintained for protocols
- passes `ConnectionManager` to ProtocolSDK to allow disconnecting from within protocol
- maintains `numPeersToUse` for each protocol within BaseProtocolSDK
* fix: pass options to protocols
* chore: update interfaces to allow public access
* chore: improve logging on protocol
* fix: renew peer upon failure
* chore(tests): allow DefaultPubsubTopic
* feat(lightpush): write peer management tests
* chore: rename test
* feat: add lock to `maintainPeers()` to handle parallelisation of requests
fixes parallelisation of lightpush.send() requests
* fix: concurrent lightpush requests
* fix: test & improve peers fetching
* chore: use getter
* address comments
* chore: smaller improvements
* feat: attempt to improve time for first lightpush.send()
* chore: use `window.interval` for type-safety
* chore: remove delays
* feat: add autoRetry
* feat: `forceUseAllPeers` to wait for all connected peers to be resoled
* fix: use pubsubTopic from current ones if not set
* fix: improve type on dial method
* enforce same pubusb on filter.subscribe, make content topic to pubsub mapping default for decoder / encoder
* fix mapping problem
* update tests
* add error handling
* fix typo
* up lock
* rm lock
* up lock
* remove only
* fix content topic
* fix ephemeral test
* fix filter unsubscribe test
* up utils
* fix subscribe test
* up interfaces and filter api
* remove only
* up ping test
* fix subscribe test
* fix push test
* fix lightPush
* fix multiple pubsub
* remove only, fix subscribe filter test
* remove only
* fix cluster ID selection and named sharding subscription test
* fix unsubscribe test
* fix light push test
* fix light push test
* fix push test
* fix relay publish
* create runNode and fix relay tests
* generalize runNodes, fix some tests
* fix store tests
* fix toAsyncIterator tests
* remove only
* fix lightPush
* use generics
* try fix test
* run failing tests
* remove only
* address failed tests, remove DefaultPubsubTopic dependency in some tests
