* feat!: do not send sync messages with empty history
A sync message without any history as no value. If there are no messages in the channel, then a sync messages does not help.
If there are messages in the channel, but this participant is not aware of them, then it can confuse other participants to assume that the channel is empty.
* fix test by adding a message to channel history
* make `pushOutgoingSyncMessage` return true even if no callback passed
* fix!: avoid SDS lamport timestamp overflow
The SDS timestamp is initialized to the current time in milliseconds, which is a 13 digits value (e.g. 1,759,223,090,052).
The maximum value for int32 is 2,147,483,647 (10 digits), which is clearly less than the timestamp.
Maximum value for uint32 is 4,294,967,295 (10 digits), which does not help with ms timestamp.
uint64 is BigInt in JavaScript, so best to be avoided unless strictly necessary as it creates complexity.
max uint64 is 18,446,744,073,709,551,615 (20 digits).
Using seconds instead of milliseconds would enable usage of uint32 valid until the year 2106.
The lamport timestamp is only initialized to current time for a new channel. The only scenario is when a user comes in a channel, and thinks it's new (did not get previous messages), and then starts sending messages. Meaning that there may be an initial timestamp conflict until the logs are consolidated, which is already handled by the protocol.
* change lamportTimestamp to uint64 in protobuf
* lamport timestamp remains close to current time
* SDS: pushOutgoingMessage is actually sync
* SDS: ensure that `ContentMessage` class is stored in local history with `valueOf` method
* feat: introduce reliable channels
Easy to use Scalable Data Sync (SDS, e2e reliability) wrapper, that includes:
- store queries upon connection to store nodes
- store queries to retrieve missing messages
* remove `channel` prefix
* attempt to improve performance when processing a lot of incoming messages
* test: split test file
* use index.ts for re-export only.
* improve if condition
* use getter for isStarted
* waku node already auto-start
* rename send
* fix lightPush.send type post rebase
* test: remove extra console.log
* SDS: emit messages as missing as soon as they are received
* make configurable elapse time for task process
* typo
* use string instead of enum for event types
* ReliableChannel.send returns the message id
* SDS: export `MessageId`
* SDS: attach retrieval hints to incoming messages
* sds: ensure items are ordered by timestamp
* test: sds: avoid using "as any" as it bypasses type checks
* test: filter: avoid using "as any" as it bypasses type checks
* test: fix tests without introducing proxy
* introduce `MessageId` type
# Conflicts:
# packages/sds/src/message_channel/message_channel.ts
* fix: own messages are not used for ack
* fix: own messages are not used for ack
* doc: long term solution is SDS protocol change
* SDS: renaming to match message function
* SDS: introduce `Message` class for easier encoding/decoding
# Conflicts:
# packages/sds/src/message_channel/events.ts
# packages/sds/src/message_channel/message_channel.ts
* SDS Message is a class now
* SDS: it's "possibly" not "partially" acknowledged.
* SDS: TODO
* SDS: fix tests
* SDS: make logs start with `waku`
* SDS: add bloom filter test
# Conflicts:
# packages/sds/src/message_channel/events.spec.ts
* SDS: improve naming
* SDS: improve naming
Messages are not "sent" or received, but pushed for processing in local queues.
* SDS: sync message should not be delivered
* SDS: renaming from earlier
* SDS: remove useless variable
* SDS: Fix comment
* SDS: sync messages do not get "delivered"
* SDS: acks
* SDS: simplify delivered event
* SDS: improve event naming
* SDS: fix comment
* SDS: make task error an official event
* SDS: Mark messages that are irretrievably lost
* SDS: remove default for irretrievable and simplify config
* SDS: typo on sync event
* SDS: add and user sender id
* SDS: resent message never get ack'd
* SDS: fix cylic dependencies
* SDS: helpful logs
* SDS: avoid duplicate history entries
* SDS: export options
* introduce `MessageId` type
* fix deprecated import
* test: own messages are not used for acks
* fix: own messages are not used for ack
* fix: own messages are not used for ack
* test: do not self-possible-ack
* doc: long term solution is SDS protocol change
- Introduce command queue system for sequential task processing
- Add comprehensive event system for message lifecycle tracking
- Restructure codebase with separate bloom_filter directory
- Export encode/decode helpers for SDS proto messages
- Use Set for deduplication in missing message detection
- Fix sync message handling for empty content messages
- Always emit MissedMessages event even with empty array
- Improve duplicate message detection logic
