Concepts are being mixed up between the global network config (static vs auto sharding), that needs to be the same of all nodes in the network, individual node configuration (eg relay node subscribing to a given shard), and the routing characteristic of a specific message (eg pubsub topic, shard).
This stops proper configuration of nwaku post 0.36.0 because we know need to be deliberate on whether nwaku nodes are running with auto or static sharding.
It also included various back and forth conversions between shards, pubsub topics, etc.
With this change, we tidy up the network configuration, and make it explicit whether it is static or auto sharded.
We also introduce the concept of routing info, which is specific to a message, and tied to the overall network configuration.
Routing info abstract pubsub topic, shard, and autosharding needs. Which should lead to easier tidy up of the pubsub concept at a later stage.
# Conflicts:
# packages/core/src/lib/connection_manager/connection_manager.ts
# packages/core/src/lib/metadata/metadata.ts
# packages/interfaces/src/metadata.ts
# packages/interfaces/src/sharding.ts
# packages/relay/src/create.ts
# packages/sdk/src/filter/filter.ts
# packages/sdk/src/filter/types.ts
# packages/sdk/src/light_push/light_push.spec.ts
# packages/tests/tests/sharding/auto_sharding.spec.ts
# packages/tests/tests/sharding/static_sharding.spec.ts
# Conflicts:
# packages/sdk/src/store/store.ts
tests: use a generator for sharded pubsub topics
set pubsub topic in encoder/decoder based on sharding type
add function for grouping content topics by pubsub topic
add autosharding config to create options
add autoshard rpc endpoints to nwaku and use in tests
set autoshard pubsub topics in all protocols
fix rebase with static sharding
removes unused function
remove console logs
remove autosharding from ShardInfo, add to EncoderOptions
fix enr and encoder/decoder options
test that same application/version hashes to same shard index
update comment on shard field
fix spelling of autosharding
fix content topic protocol in tests
add sharding type alias and function to determine topic in encoders/decoders
move DefaultPubsubTopic from core to interfaces
* `ProtocolCreateOptions` now has `pubSubTopic` as `pubSubTopic[]`
* chore: update encoder & decoder to support `PubSubTopic`
* feat(protocols): allow multiple `PubSubTopic[]`
* feat(relay): allow multiple `PubSubTopic[]`
* chore(tests): update for new API
* chore: minor fixes
* chore: make store more robust
* fix(relay): correctly set types
* chore(address comments): update terminology around configured pubsub topics
* chore(address comments): minor refactoring
* chore(relay): split `subscribe` into smaller functions for readability & modularity
* chore(address comments): refactor `waitForGossipSubPeerInMesh`
* chore(store): only allow to query one `pubSubTopic`
* fix: `store` bug
* feat(tests): add some basic tests
* sharding utils
* address comments
* feat(relay): re-add API for `getMeshPeers`
* update error message
Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>
* refactor for new API
* feat: simplify handling of observers (#1614)
* refactor: simplify handling of observers
* refactor: Remove redundant PubSubTopic from Observer
* use `??` instead of `||`
* update `pubsubTopic` to `pubSubTopic`
* update `interval` typo
* change occurence of `pubsubTopic` to `pubSubTopic`
* relay: rm `getAllMeshPeers` and make `pubSubTopics` public
* relay: use `push_or_init_map` and move to `utils`
* fix: update API for tests
* fix: relay waitForRemotePeer
---------
Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>
