These are high level configuration values, that need to be consistent within a given execution. Setting them as default value at such a low level is a footgun risk.
Concepts are being mixed up between the global network config (static vs auto sharding), that needs to be the same of all nodes in the network, individual node configuration (eg relay node subscribing to a given shard), and the routing characteristic of a specific message (eg pubsub topic, shard).
This stops proper configuration of nwaku post 0.36.0 because we know need to be deliberate on whether nwaku nodes are running with auto or static sharding.
It also included various back and forth conversions between shards, pubsub topics, etc.
With this change, we tidy up the network configuration, and make it explicit whether it is static or auto sharded.
We also introduce the concept of routing info, which is specific to a message, and tied to the overall network configuration.
Routing info abstract pubsub topic, shard, and autosharding needs. Which should lead to easier tidy up of the pubsub concept at a later stage.
# Conflicts:
# packages/core/src/lib/connection_manager/connection_manager.ts
# packages/core/src/lib/metadata/metadata.ts
# packages/interfaces/src/metadata.ts
# packages/interfaces/src/sharding.ts
# packages/relay/src/create.ts
# packages/sdk/src/filter/filter.ts
# packages/sdk/src/filter/types.ts
# packages/sdk/src/light_push/light_push.spec.ts
# packages/tests/tests/sharding/auto_sharding.spec.ts
# packages/tests/tests/sharding/static_sharding.spec.ts
# Conflicts:
# packages/sdk/src/store/store.ts
For an edge node, there is no such thing as a "pubsub topic configuration". An edge node should be able to operate for any possible shard, and it is a per-protocol matter (eg send message with light push).
A relay node do subscribe to shards, but in this case, even metadata protocol does not need to advertise them, this is already handled by gossipsub.
Only service node should advertise their shards via metadata protocol, which is out of scope for js-waku.
# Conflicts:
# packages/interfaces/src/connection_manager.ts
* add FF for auto recovery
* implement connection locking, connection maintenance, auto recovery, bootstrap connections maintenance and fix bootstrap peers dropping
* add ut for peer manager changes
* implement UT for Connection Limiter
* increase connection maintenance interval
* update e2e test
* implement new peer manager, use in lightPush, improve retry manager and fix retry bug
* fix unsubscribe issue
* remove not needed usage of pubsub, use peer manager in store sdk
* chore: remove deprecated filter implementation
* update tests
* update next filter for new peer manager
* skip IReceiver test, remove unused utility
* remove comment
* fix typo
* remove old connection based peer manager
* update types, export, and edge case for light push
* add retry manager tests
* add new peer manager tests
* refactor tests
* use peer manager events in filter and check for pubsub topic as well
* update test names
* address comments
* unskip Filter e2e test
* address more comments, remove duplication
* skip CI test
* update after merge
* move to peer:idenfity and peer:disconnect events, improve mapping in filter subscriptions
* update tests
* add logs and change peer manager time lock to 10s
* feat: implement shard retrieval for store and improve set store peers usage
* remove log
* remove only, improve condition
* implement smarter way to retrieve peers
* up tests
* update mock
* address nits, add target to eslint, revert to es2022
* remove IBaseProtocol
* fix references, interfaces and integration
* fix ci
* up mock
* up lock
* add mock for local storage
* add missing prop, fix tests
* up lock
- Remove all console.log statements (8 instances)
- Update type usage from DecodedMessage[] to IDecodedMessage[]
- Replace assert.equal() with expect().to.equal() for consistency
- Import expect from chai alongside existing imports
These changes address all feedback from PR review without altering test logic.
* create new filter api
* implement await on main methods on new Filter
* add info logs in new filter
* add logs to subscription impl
* remove lint supress
* add unit tests
* introduce E2E tests
* update e2e tests and add case for testing filter recovery after nwaku nodes replacement
* add new test cases for max limits and enable decoders as array on new filter
* fix edge case testing, correct test cases
* skip test
* update error message
* up text
* up text
* fix lint
* implement unsubscribeAll
* add js-dock to new filter
* add cspell
* implement TTL set for message history
* update Filter test suit, make service nodes connected to each other, remove single node Filter test suit, use 0.35 nwaku image
* update light push tests
* improve auto shard tests
* update static sharding test
* skip blocked tests
* fix test
* remove usage of pusubtopic with nwaku
* remove comment
The waku test fleet is unstable and should not be used as boostrap.
The issue is that DNS Discovery looks for "2" nodes and stop there. Meaning that if those "discovered" nodes are not reachable (ie, coming from test suite). js-waku does not connect.
* doc: build first to avoid not found deps issues
* fix: Remove XMLHttpRequest usage
XMLHttpRequest API is deprecated and not available in browser extensions.
Replace the culprit dependency with a more modern one.
Some options are removed. The assumption is that nobody uses them. It can always be added if a developer wants the flexibility.
* test: simplify test
The test focus on testing DNS Discovery, there is no need to also have a nwaku local node for bootstrap.
Bootstrap on nwaku local node is used in many other tests.
* create retry manager
* update tests
* add retry manager tests, update peer manager
* fix start & merge with master
* return send to many logic
* add new error handling
* add sections to protocol errors
* fix check
* up test
* add waku.start in test
* fix check and test
* improve name
* chore: upgrade nwaku to v0.33.1
* chore: upgrade to nwaku 0.34.0
* feat: connect nwaku nodes amongst each other over relay
* chore(lightpush): use multiple service nodes for lightpush (instead of just one)
- nwaku now expects >=1 nodes at least connected
* chore: all single-node lightpush requests should now be expected to fail
* chore: update sharding tests
* chore: update tests
* chore: improve Docker network config reliability
* chore: deduplicate ecies encrypted payloads
* chore: update to precise expects
* fix: return early if expect passes
* chore: lightpush 5 times instead of 30
* fix: non duplicacy should happen in application-specific scenario
* chore: update mocha config + fix epehermal tests
* chore: reinstall deps after rebase
* chore: attempt stability for test suite
* fix: store tests to now use multiple nodes, delete uneeded test
* fix: memory leak
* chore: switch while loop with timeout-promise
* chore: remove redundant nodes startup
* chore: add delays for nwaku setup
* up lock
* make ConnectionManager use ctor
* reform connection manager configurations
* remove log param from peerManager
* make PeerManager use only ConnectionManager, move getPeers to ConnectionManager, remove not needed code
* remove allPeers and connectedPeers from BaseProtocolCore, update tests, add getPeers for IWaku
* use only one peerManager from Waku object
* remove IBaseProtocolSDK and merge with PeerManager
* re-implement peerManager, remove ProtocolUseOptions
* remove not needed test, up lock
* update deps and lock
* remove old test for peerManager, fix check and spell
* rename to getConnectedPeers
* feat: improve filter subscriptions (#2193)
* add message cache to Filter
* remove WakuOptions and use only ProtocolCreateOptions
* move subscribe options to createLightNode Fitler protocol options
* rename SubscriptionManager to Subscription
* rename to CreateNodeOptions
* add warning
* feat: introduce subscription manager (#2202)
* feat: inroduce subscription manager
* fix: make pipeline succeed (#2238)
* fix test
* use hardcoded value
* update playwright
* fix test:browser
* up lock
* make peer retrieval probabilistic
* add comments
* up lightpush tests
* add tests for peer_manager, improve folder structure
* create named files for protocols
* create named files, simplify project structure
* remove only
We are in the process of deprecating pubsub-topic param from nim-waku.
The pubsub-topic shouldn't be used to start nim-waku and instead, a
combination of 1 cluster-id plus n shard should be given. This aims to
start using the new added param: shard.
