payload & transfer

standards/core/sync.md

@@ -7,19 +7,25 @@ contributors:
  - Hanno Cornelius <hanno@status.im>
 ---
 
-## Abstract
+# Abstract
-This specification explains the `WAKU-SYNC` protocol
+This specification explains `WAKU-SYNC`
-which enables the reconciliation of two sets of message hashes
+which enables the syncronization of messages between 2 Store nodes.
-in the context of keeping multiple Store nodes synchronized.
 
-## Specification
+# Specification
 
-**Protocol identifier**: `/vac/waku/reconciliation/1.0.0`
+Waku Sync consists of 2 protocols; reconciliation and transfer.
+Reconciliation is the process of finding differences in 2 sets of message hashes.
+Transfer is then used to bilateraly send messages to the other peer.
+The end goal being that both peers have the same set of hashes and messages.
 
 #### Terminology
 The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, 
 “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt).
 
+## Reconciliation
+
+**Libp2p Protocol identifier**: `/vac/waku/reconciliation/1.0.0`
+
 #### Message Ids
 Message Ids MUST be composed of the timestamp and the hash of the Waku messages.
 
@@ -39,16 +45,16 @@ The first bound MUST be strictly smaller than the second one.
 
 #### Range Fingerprinting
 The fingerprint of a range MUST be the XOR operation applied to
-all the hashes of the Ids included in that range.
+all the hashes of the messages included in that range.
 
 #### Range Type
 Every range MUST have one of the following types; skip, fingerprint or item set.
 
 - Skip type is used to signal already processed ranges that MUST be ignored.
-- Fingerprint type signify that this range fingerprint MUST be compared when received.
+- Fingerprint type signify that fingerprints MUST be compared when received.
 - Item set type contain multiple message Ids that MUST all be compared when received.
 > Item sets are an optimization, stopping the recursion early can
-save many network roundtrips.
+save network roundtrips.
 
 #### Range Processing
 Ranges have to be processed differently acording to their types.
@@ -69,7 +75,7 @@ the same as the exclusive upper bounds of the previous range or zero.
 To achieve this, it MAY be needed to add skip ranges.
 > For example, a skip range can be added with
 an exclusive upper bound equal to the first range lower bound.
-This way the receiving peer knows to ignore the range from zero to the start of the ranges
+This way the receiving peer knows to ignore the range from zero to the start of the sync window.
 
 Every timestamps after the first MUST be noted as the difference from the previous one.
 If the timestamp is the same, zero MUST be used and the hash MUST be added.
@@ -85,7 +91,51 @@ The added hash MUST be trucated up to and including the first differetiating byt
 #### Varints
 TODO
 
-## Implementation
+#### Payload encoding
+The wire level payload MUST be encoded as follow.
+> The & denote concatenation
+
+1. varint bytes of the delta encoded timestamp &
+2. if timestamp is zero, delta encoded hash bytes  &
+3. 1 byte, the range type &
+4. either
+    - 32 bytes fingerprint &
+    - varint bytes of the item set length & bytes of every items &
+    - if skip range, nothing
+
+5. repeat 1 to 4 for all ranges
+
+## Transfer Protocol
+
+**Libp2p Protocol identifier**: `/vac/waku/transfer/1.0.0`
+
+TODO
+
+should not accept messages from peers not being syncing with.
+
+should send message as soon as a diff is found.
+
+in the future verify RLN proof of messages.
+
+### Wire specification
+```protobuf
+syntax = "proto3";
+
+package waku.sync.v2;
+
+import "waku/message/v1/message.proto";
+
+message WakuMessageAndTopic {
+  // Full message content and associated pubsub_topic as value
+  optional waku.message.v1.WakuMessage message = 1;
+  optional string pubsub_topic = 2;
+}
+```
+
+# Implementation
+The flexibitity of the protocol implies that much is left to the implementers.
+What will follow is NOT part of the specification.
+This section was created to inform implementations.
 
 #### Parameters 
 #TODO fix copy pasta from research issue
@@ -95,34 +145,26 @@ T -> Item set threshold. If a range length is <= than T, all items are sent. Hig
 B -> Partitioning count. When recursively splitting a range, it is split into B sub ranges. Higher B reduce round trips at the cost of computing more fingerprints.
 
 #### Storage
-TODO
+The storage implementation should reflect the context.
-
+Most messages that will be added will be recent and
-A local cache of message Ids MUST be maintained when the node is online.
+removed messages will be older ones.
-This storage MUST keep Ids ordered at all times.
+When differences are found some messages will have to be inserted randomly.
-
+It is expected to be a less likely case than time based insertion and removal.
-This storage is critical for the various function of the protocol and should as efficient as possible.
+Last but not least it must be optimized for fingerprinting
-How this storage is implemented however, is outside the scope of this specification.
+as it is the most often used operation.
 
 TODO mention trees vs arrays???
 
-The storage implementation should reflect the Waku context.
+#### Sync Window
-Most messages that will be added will be recent and
+TODO rephrase
-all removed messages will be older ones.
-When differences are found some messages will have to be inserted randomly.
-It is expected to be a less likely case than time based insertion and removal.
-Last but not least it must be optimized for sequential read
-as it is the most often used operation.
 
-#### Range
+We also offset the sync window by 20 seconds in the past.
-TODO
-
-We also offset the sync range by 20 seconds in the past.
 The actual start of the sync range is T-01:00:20 and the end T-00:00:20
 This is to handle the inherent jitters of GossipSub.
 In other words, it is the amount of time needed to confirm if a message is missing or not.
 
-#### Interval
+#### Sync Interval
-TODO
+TODO rephrase
 
 Ad-hoc syncing can be useful in some cases but continuous periodic sync
 minimize the differences in messages stored across the network.
@@ -130,7 +172,7 @@ Syncing early and often is the best strategy.
 The default used in nwaku is 5 minutes interval between sync with a range of 1 hour.
 
 #### Peer Choice
-TODO
+TODO rephrase
 
 Peering strategies can lead to inadvertently segregating peers and reduce sampling diversity.
 We randomly select peers to sync with for simplicity and robustness.
@@ -151,5 +193,6 @@ Copyright and related rights waived via
 [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
 
 ## References
- - https://logperiodic.com/rbsr.html
+ - [RBSR](https://github.com/AljoschaMeyer/rbsr_short/blob/main/main.pdf)
- - https://github.com/hoytech/negentropy
+ - [Negentropy Explainer](https://logperiodic.com/rbsr.html)
+ - [Master Thesis](https://github.com/AljoschaMeyer/master_thesis/blob/main/main.pdf)