Concepts are being mixed up between the global network config (static vs auto sharding), that needs to be the same of all nodes in the network, individual node configuration (eg relay node subscribing to a given shard), and the routing characteristic of a specific message (eg pubsub topic, shard).
This stops proper configuration of nwaku post 0.36.0 because we know need to be deliberate on whether nwaku nodes are running with auto or static sharding.
It also included various back and forth conversions between shards, pubsub topics, etc.
With this change, we tidy up the network configuration, and make it explicit whether it is static or auto sharded.
We also introduce the concept of routing info, which is specific to a message, and tied to the overall network configuration.
Routing info abstract pubsub topic, shard, and autosharding needs. Which should lead to easier tidy up of the pubsub concept at a later stage.
# Conflicts:
# packages/core/src/lib/connection_manager/connection_manager.ts
# packages/core/src/lib/metadata/metadata.ts
# packages/interfaces/src/metadata.ts
# packages/interfaces/src/sharding.ts
# packages/relay/src/create.ts
# packages/sdk/src/filter/filter.ts
# packages/sdk/src/filter/types.ts
# packages/sdk/src/light_push/light_push.spec.ts
# packages/tests/tests/sharding/auto_sharding.spec.ts
# packages/tests/tests/sharding/static_sharding.spec.ts
# Conflicts:
# packages/sdk/src/store/store.ts
* update Filter test suit, make service nodes connected to each other, remove single node Filter test suit, use 0.35 nwaku image
* update light push tests
* improve auto shard tests
* update static sharding test
* skip blocked tests
* fix test
* remove usage of pusubtopic with nwaku
* remove comment
* create retry manager
* update tests
* add retry manager tests, update peer manager
* fix start & merge with master
* return send to many logic
* add new error handling
* add sections to protocol errors
* fix check
* up test
* add waku.start in test
* fix check and test
* improve name
* chore: upgrade nwaku to v0.33.1
* chore: upgrade to nwaku 0.34.0
* feat: connect nwaku nodes amongst each other over relay
* chore(lightpush): use multiple service nodes for lightpush (instead of just one)
- nwaku now expects >=1 nodes at least connected
* chore: all single-node lightpush requests should now be expected to fail
* chore: update sharding tests
* chore: update tests
* chore: improve Docker network config reliability
* chore: deduplicate ecies encrypted payloads
* chore: update to precise expects
* fix: return early if expect passes
* chore: lightpush 5 times instead of 30
* fix: non duplicacy should happen in application-specific scenario
* chore: update mocha config + fix epehermal tests
* chore: reinstall deps after rebase
* chore: attempt stability for test suite
* fix: store tests to now use multiple nodes, delete uneeded test
* fix: memory leak
* chore: switch while loop with timeout-promise
* chore: remove redundant nodes startup
* chore: add delays for nwaku setup
* up lock
* make ConnectionManager use ctor
* reform connection manager configurations
* remove log param from peerManager
* make PeerManager use only ConnectionManager, move getPeers to ConnectionManager, remove not needed code
* remove allPeers and connectedPeers from BaseProtocolCore, update tests, add getPeers for IWaku
* use only one peerManager from Waku object
* remove IBaseProtocolSDK and merge with PeerManager
* re-implement peerManager, remove ProtocolUseOptions
* remove not needed test, up lock
* update deps and lock
* remove old test for peerManager, fix check and spell
* rename to getConnectedPeers
* feat: improve filter subscriptions (#2193)
* add message cache to Filter
* remove WakuOptions and use only ProtocolCreateOptions
* move subscribe options to createLightNode Fitler protocol options
* rename SubscriptionManager to Subscription
* rename to CreateNodeOptions
* add warning
* feat: introduce subscription manager (#2202)
* feat: inroduce subscription manager
* fix: make pipeline succeed (#2238)
* fix test
* use hardcoded value
* update playwright
* fix test:browser
* up lock
* make peer retrieval probabilistic
* add comments
* up lightpush tests
* add tests for peer_manager, improve folder structure
* create named files for protocols
* create named files, simplify project structure
* remove only
We are in the process of deprecating pubsub-topic param from nim-waku.
The pubsub-topic shouldn't be used to start nim-waku and instead, a
combination of 1 cluster-id plus n shard should be given. This aims to
start using the new added param: shard.
* fix comment of default number of peers
* export default number of peers from base protocol sdk
* rename to light_push, move class to separate file
* move waitForRemotePeer to sdk package
* add todo to move waitForGossipSubPeerInMesh into @waku/relay
* clean up waitForRemotePeer, split metadata await from event and optimise, decouple from protocol implementations
* simplify and rename ILightPush interface
* use only connected peers in light push based on connections instead of peer renewal mechanism
* improve readability of result processing in light push
* fix check & update tests
* address tests, add new test cases, fix racing condition in StreamManager
* use libp2p.getPeers
* feat: confirm metadata and protocols needed in waitForRemotePeer
* rely on passed protocols and fallback to mounted
* add I prefix to Waku interface
* implement waku.connect method
* add docs to IWaku interface
* remove export and usage of waitForRemotePeer
* move wait for remote peer related to Realy out of @waku/sdk
* change tests to use new API
* fix linting
* update size limit
* rename .connect to .waitForPeer
* export waitForRemotePeer and mark as deprecated
* feat: add mocha tests to @waku/sdk and cover waitForRemotePeer (#2163)
* feat: add mocha tests to @waku/sdk and cover waitForRemotePeer
* add waitForRemote UTs
* remove junk
* feat: expose peerId and protocols from WakuNode (#2166)
* chore: expose peerId and protocols from WakuNode
* remove unused method
* move to private method
* rename to waitForPeers
* up test
* fix comment of default number of peers
* export default number of peers from base protocol sdk
* rename to light_push, move class to separate file
* move waitForRemotePeer to sdk package
* add todo to move waitForGossipSubPeerInMesh into @waku/relay
* clean up waitForRemotePeer, split metadata await from event and optimise, decouple from protocol implementations
* simplify and rename ILightPush interface
* use only connected peers in light push based on connections instead of peer renewal mechanism
* improve readability of result processing in light push
* fix check & update tests
* address tests, add new test cases, fix racing condition in StreamManager
* use libp2p.getPeers
* feat: confirm metadata and protocols needed in waitForRemotePeer
* rely on passed protocols and fallback to mounted
* imrpove iteration for existing connections
* address protocol adverisement in CI
* add protocols needed
* add missing protocols
* make lightpush and filter default for tests
* up
* fix comment of default number of peers
* export default number of peers from base protocol sdk
* rename to light_push, move class to separate file
* move waitForRemotePeer to sdk package
* add todo to move waitForGossipSubPeerInMesh into @waku/relay
* clean up waitForRemotePeer, split metadata await from event and optimise, decouple from protocol implementations
* simplify and rename ILightPush interface
* use only connected peers in light push based on connections instead of peer renewal mechanism
* improve readability of result processing in light push
* fix check & update tests
* address tests, add new test cases, fix racing condition in StreamManager
* use libp2p.getPeers
* chore: upgrade libp2p and related packages
* fix: public key on peer id can now be undefined
* chore: upgrade remainder packages
* chore: fix tests
* chore: address comments
* chore: upgrade playwright and proto
* feat: introduce HealthManager
* feat: make health accessible on Waku object
* feat: update health from protocols
* chore: add access modifiers to healthmanager
* feat: use a HealthManager singleton
* chore: add tests for Filter, LightPush and Store
* feat: add overall node health
* chore: update protocol health to consider Store protocol
* chore: setup generic test utils instead of using filter utils
* tests: add a health status matrix check from 0-3
* chore: increase timeout for failing tests in CI
tests pass locally without an increased timeout, but fail in CI
* chore: move name inference to HealthManager
* tests: abstract away node creation and teardown utils
* fix: import
* fix: use pubsubTopic from current ones if not set
* fix: improve type on dial method
* enforce same pubusb on filter.subscribe, make content topic to pubsub mapping default for decoder / encoder
* fix mapping problem
* update tests
* add error handling
* fix typo
* up lock
* rm lock
* up lock
* remove only
* fix content topic
* fix ephemeral test
* fix filter unsubscribe test
* up utils
* fix subscribe test
* up interfaces and filter api
* remove only
* up ping test
* fix subscribe test
* fix push test
* fix lightPush
* fix multiple pubsub
* remove only, fix subscribe filter test
* remove only
* fix cluster ID selection and named sharding subscription test
* fix unsubscribe test
* fix light push test
* fix light push test
* fix push test
* fix relay publish
* create runNode and fix relay tests
* generalize runNodes, fix some tests
* fix store tests
* fix toAsyncIterator tests
* remove only
* fix lightPush
* use generics
* try fix test
* run failing tests
* remove only
* address failed tests, remove DefaultPubsubTopic dependency in some tests
* move relay related code
* move libp2p to utils
* define CreateWakuNodeOptions
* improve options
* make libp2p use from create function
* add bootstrapPeers option
* fix lint
* fix types, imports
* fix exports
* use bootstrap along default bootstrap
* fix test as REST does not return peer though connection is made
* rollback condition change
* enable gossipSub back for every node
This commit modifies functions in ServiceNode to use the REST API
instead of JSON RPC when reading messages for a pubsub topic or
content topic, and when ensuring a nwaku node is subscribed to a
pubsub topic. Also modifies default Docker params to enable the
rest API and provide a port.
* use graceful timeout mechanism
* set max timeout for all hooks
* small fix
* small fix
* use MOCHA_HOOK_MAX_TIMEOUT as default timeoutDuration
* force retry even when the hook fails
* use custom hooks
* fix global timeout problem
* fix unwanted change
* fix enr issue
* force retry on before error as well
* append `p2p` with the multiaddrs from ENR
* fix(tests): add p2p checks for ENR multiaddrs
* fix(tests): type getter
* chore(ci): remove debug flag from nwaku_master and go-waku tests
