feat: waku sync 2.0

standards/core/sync.md

@@ -7,126 +7,204 @@ contributors:
  - Hanno Cornelius <[email protected]>
 ---
 
-## Abstract
-This specification explains the `WAKU-SYNC` protocol
-which enables the reconciliation of two sets of message hashes
-in the context of keeping multiple Store nodes synchronized.
-Waku Sync is a wrapper around
-[Negentropy](https://github.com/hoytech/negentropy) a [range-based set reconciliation protocol](https://logperiodic.com/rbsr.html).
+# Abstract
+This specification explains `WAKU-SYNC`
+which enables the synchronization of messages between nodes storing sets of [`14/WAKU2-MESSAGE`](https://rfc.vac.dev/waku/standards/core/14/message)
 
-## Specification
+# Specification
 
-**Protocol identifier**: `/vac/waku/sync/1.0.0`
+Waku Sync consists of 2 libp2p protocols; reconciliation and transfer.
+The Reconciliation protocol finds differences in sets of messages.
+The Transfer protocol is used to exchange the differences found with other peers.
+The end goal being that peers have the same set of messages.
 
-### Terminology
+#### Terminology
 The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, 
 “RECOMMENDED”, “MAY”, and “OPTIONAL” in this document are to be interpreted as described in [RFC2119](https://www.ietf.org/rfc/rfc2119.txt).
 
-The term Negentropy refers to the protocol of the same name.
-Negentropy payload refers to
-the messages created by the Negentropy protocol.
-Client always refers to the initiator
-and the server the receiver of the first payload.
-
-### Design Requirements
-Nodes enabling Waku Sync SHOULD
-manage and keep message hashes in a local cache
-for the range of time
-during which synchronization is required.
-Nodes SHOULD use the same time range,
-for Waku we chose one hour as the global default.
-Waku Relay or Light Push protocol MAY be enabled
-and used in conjunction with Sync
-as a source of new message hashes
-for the cache.
-
-Nodes MAY use the Store protocol
-to request missing messages once reconciliation is complete
-or to provide messages to requesting clients.
-
-### Payload
+## Reconciliation
 
+**Libp2p Protocol identifier**: `/vac/waku/reconciliation/1.0.0`
+
+The protocol finds differences between 2 peers by
+comparing _fingerprints_ of _ranges_ of message _IDs_.
+_Ranges_ are encoded into payloads, exchanged between the peers and when the range _fingerprints_ are different, splitted into smaller ones.
+This process repeats until _ranges_ include a small number of messages
+at this point _IDs_ are sent for comparison instead of _fingerprints_.
+
+#### Overview
+
+1. The requestor choose a time range to sync.
+2. The range is encoded into a payload and sent.
+3. The requestee receive the payload and decode it.
+4. The range is processed and more ranges produced.
+5. The new ranges are encoded and sent.
+6. Payloads are repeatedly exchanged and differences between the peers are discovered.
+7. The synchronization is done when no ranges are left to process.
+
+#### Message Ids
+Message _IDs_ MUST be composed of the timestamp and the hash of the [`14/WAKU2-MESSAGE`](https://rfc.vac.dev/waku/standards/core/14/message).
+
+The timestamp MUST be the time of creation and
+the hash MUST follow the
+[deterministic message hashing specification](https://rfc.vac.dev/waku/standards/core/14/message#deterministic-message-hashing)
+
+> This way the message Ids can always be totally ordered,
+first chronologically according to the timestamp and then
+disambiguate based on the hash lexical order
+in cases where the timestamp is the same.
+
+#### Range Bounds
+A _range_ MUST consists of 2 _IDs_, the first bound is
+inclusive the second bound exclusive.
+The first bound MUST be strictly smaller than the second one.
+
+#### Range Fingerprinting
+The _fingerprint_ of a range MUST be the XOR operation applied to
+the hash of all message _IDs_ included in that _range_.
+
+#### Range Type
+Every _range_ MUST have one of the following types; _fingerprint_, _skip_ or _item set_.
+
+- _Fingerprint_ type contains a _fingerprint_.
+- _Skip_ type contains nothing and is used to signal already processed _ranges_.
+- _Item set_ type contains message _Ids_ and a _resolved_ boolean.
+> _Item sets_ are an optimization, sending multiple _IDs_ instead of
+recursing further reduce the number of round-trips.
+
+#### Range Processing
+_Ranges_ have to be processed differently according to their types.
+
+- **Equal** _fingerprint ranges_ MUST become _skip ranges_.
+- **Unequal** _fingerprint ranges_ MUST be splitted into smaller ranges. The new type MAY be either _fingerprint_ or _item set_.
+- **Unresolved** _item set_ ranges MUST be checked for differences and marked resolved.
+- **Resolved** _item set_ ranges MUST be checked for differences and become skip ranges.
+- _Skip_ ranges MUST be merged with other consecutive _skip ranges_.
+
+In the case where only skip ranges remains, the synchronization is done.
+
+### Delta Encoding
+_Ranges_ and timestamps MUST be delta encoded as follows for efficient transmission.
+
+All _ranges_ to be transmitted MUST be ordered and only upper bounds used.
+> Inclusive lower bounds can be omitted because they are always
+the same as the exclusive upper bounds of the previous range or zero.
+
+To achieve this, it MAY be needed to add _skip ranges_.
+> For example, a _skip range_ can be added with
+an exclusive upper bound equal to the first range lower bound.
+This way the receiving peer knows to ignore the range from zero to the start of the sync time window.
+
+Every _ID_'s timestamps after the first MUST be noted as the difference from the previous one.
+If the timestamp is the same, zero MUST be used and the hash MUST be added.
+The added hash MUST be truncated up to and including the first differentiating byte.
+
+| Timestamp | Hash | Timestamp (encoded) | Hash (encoded) 
+| - | - | - | -
+| 1000 | 0x4a8a769a... | 1000 | -
+| 1002 | 0x351c5e86... | 2 | -
+| 1002 | 0x3560d9c4... | 0 | 0x3560
+| 1003 | 0xbeabef25... | 1 | -
+
+#### Varints
+All _varints_ MUST be little-endian base 128 variable length integers (LEB128) and minimally encoded.
+
+#### Payload encoding
+The wire level payload MUST be encoded as follow.
+> The & denote concatenation.
+
+1. _varint_ bytes of the delta encoded timestamp &
+2. if timestamp is zero, 1 byte for the hash bytes length & the hash bytes &
+3. 1 byte, the _range_ type &
+4. either
+    - 32 bytes _fingerprint_ &
+    - _varint_ bytes of the item set length & bytes of every items &
+    - if _skip range_, do nothing
+
+5. repeat steps 1 to 4 for all ranges.
+
+## Transfer Protocol
+
+**Libp2p Protocol identifier**: `/vac/waku/transfer/1.0.0`
+
+The transfer protocol SHOULD send messages as soon as
+a difference is found via reconciliation.
+It MUST only accept messages from peers the node is reconciliating with.
+New message IDs MUST be added to the reconciliation protocol.
+The payload sent MUST follow the wire specification below.
+
+### Wire specification
 ```protobuf
 syntax = "proto3";
 
-package waku.sync.v1;
+package waku.sync.transfer.v1;
 
-message SyncPayload {
-  optional bytes negentropy = 1;
+import "waku/message/v1/message.proto";
 
-  repeated bytes hashes = 20;
+message WakuMessageAndTopic {
+  // Full message content and associated pubsub_topic as value
+  optional waku.message.v1.WakuMessage message = 1;
+  optional string pubsub_topic = 2;
 }
 ```
 
-### Session Flow
-A client initiates a session with a server
-by sending a `SyncPayload` with
-only the `negentropy` field set.
-This field MUST contain
-the first negentropy payload
-created by the client
-for this session.
-
-The server receives a `SyncPayload`.
-A new negentropy payload is computed from the received one.
-The server sends back a `SyncPayload` to the client.
-
-The client receives a `SyncPayload`.
-A new negentropy payload OR an empty one is computed.
-If a new payload is computed then
-the exchanges between client and server continues until
-the client computes an empty payload.
-This client computation also outputs any hash differences found,
-those MUST be stored.
-In the case of an empty payload,
-the reconciliation is done,
-the client MUST send back a `SyncPayload`
-with all the missing server hashes in the `hashes` field and
-an empty `nengentropy` field.
-
-## Attack Vectors
-Nodes using `WAKU-SYNC` are fully trusted.
-Message hashes are assumed to be of valid messages received via Waku Relay or Light push.
+# Implementation
+The flexibility of the protocol implies that much is left to the implementers.
+What will follow is NOT part of the specification.
+This section was created to inform implementations.
 
-Further refinements to the protocol are planned
-to reduce the trust level required to operate.
-Notably by verifying messages RLN proof at reception.
+#### Parameters 
+Two useful parameters to add to your implementation are partitioning count and the item set threshold.
 
-## Implementation
-The following is not part of the specifications but good to know implementation details.
+The partitioning count is the number of time a range is splitted.
+Higher value reduce round trips at the cost of computing more fingerprints.
 
-### Peer Choice
-Peering strategies can lead to inadvertently segregating peers and reduce sampling diversity.
-We randomly select peers to sync with for simplicity and robustness.
+The threshold for which item sets are sent instead of fingerprints.
+Higher value sends more items which means higher chance of duplicates but
+reduce the amount of round trips overall.
 
-A good strategy can be devised but we chose not to.
+#### Storage
+The storage implementation should reflect the context.
+Most messages that will be added will be recent and
+removed messages will be older ones.
+When differences are found some messages will have to be inserted randomly.
+It is expected to be a less likely case than time based insertion and removal.
+Last but not least it must be optimized for fingerprinting
+as it is the most often used operation.
 
-### Interval
+#### Sync Interval
 Ad-hoc syncing can be useful in some cases but continuous periodic sync
 minimize the differences in messages stored across the network.
 Syncing early and often is the best strategy.
-The default used in nwaku is 5 minutes interval between sync with a range of 1 hour.
+The default used in Nwaku is 5 minutes interval between sync with a range of 1 hour.
 
-### Range
-We also offset the sync range by 20 seconds in the past.
-The actual start of the sync range is T-01:00:20 and the end T-00:00:20
+#### Sync Window
+By default we offset the sync window by 20 seconds in the past.
+The actual start of the sync range is T-01:00:20 and the end T-00:00:20 in most cases.
 This is to handle the inherent jitters of GossipSub.
 In other words, it is the amount of time needed to confirm if a message is missing or not.
 
-### Storage
-The storage implementation should reflect the Waku context.
-Most messages that will be added will be recent and
-all removed messages will be older ones.
-When differences are found some messages will have to be inserted randomly.
-It is expected to be a less likely case than time based insertion and removal.
-Last but not least it must be optimized for sequential read
-as it is the most often used operation.
+#### Peer Choice
+Wrong peering strategies can lead to inadvertently segregating peers and
+reduce sampling diversity.
+Nwaku randomly select peers to sync with for simplicity and robustness.
+
+More sophisticated strategies may be implemented in future.
+
+## Attack Vectors
+Nodes using `WAKU-SYNC` are fully trusted.
+Message hashes are assumed to be of valid messages received via Waku Relay or Light push.
+
+Further refinements to the protocol are planned
+to reduce the trust level required to operate.
+Notably by verifying messages RLN proof at reception.
 
 ## Copyright
 
 Copyright and related rights waived via
 [CC0](https://creativecommons.org/publicdomain/zero/1.0/).
 
 ## References
- - https://logperiodic.com/rbsr.html
- - https://github.com/hoytech/negentropy
+ - [RBSR](https://github.com/AljoschaMeyer/rbsr_short/blob/main/main.pdf)
+ - [Negentropy Explainer](https://logperiodic.com/rbsr.html)
+ - [Master Thesis](https://github.com/AljoschaMeyer/master_thesis/blob/main/main.pdf)