From ccb9ae7f44e32ff9778bdbe0ed02a56e0ff29996 Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Mon, 30 Sep 2019 13:08:04 +0800 Subject: [PATCH 01/10] Add 0.1.0 remote log initial spec --- remote-log.md | 216 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 remote-log.md diff --git a/remote-log.md b/remote-log.md new file mode 100644 index 00000000..9f0f0f4d --- /dev/null +++ b/remote-log.md @@ -0,0 +1,216 @@ +# Remote log specification + +> Version: 0.1.0 (Draft) +> +> Authors: Oskar Thorén oskar@status.im, Dean Eigenmann dean@status.im + +## Table of Contents + +- [Abstract](#abstract) +- [Definitions](#definitions) +- [Wire Protocol](#wire-protocol) + - [Secure Transport, storage, and name system](#secure-transport-storage-and-name-system) + - [Payloads](#payloads) +- [Synchronization](#synchronization) + - [Roles](#roles) + - [Flow](#flow) + - [Remote log](#remote-log) + - [Next page semantics](#next-page-semantics) + - [Interaction with MVDS](#interaction-with-mvds) +- [Footnotes](#footnotes) +- [Acknowledgements](#acknowledgements) + +## Abstract + +A remote log is a replication of a local log. This means a node can read data from a node that is offline. + +This specification is complemented by a proof of concept implementation 1. + +## Definitions + +| Term | Definition | +| ----------- | -------------------------------------------------------------------------------------- | +| CAS | Content-addressed storage. Stores data that can be addressed by its hash. | +| NS | Name system. Associates mutable data to a name. | +| Remote log | Replication of a local log at a different location. | + +## Wire Protocol + +### Secure Transport, storage, and name system + +This specification does not define anything related to to: secure transport, +content addressed storage, or the name system. It is assumed these capabilities +are abstracted away in such a way that any such protocol can easily be +implemented. + + + +### Payloads + +Payloads are implemented using [protocol buffers v3](https://developers.google.com/protocol-buffers/). + +**CAS service**: + +```protobuf +package vac.cas; + +service CAS { + rpc Add(Content) returns (Address) {} + rpc Get(Address) returns (Content) {} +} + +message Address { + bytes id = 1; +} + +message Content { + bytes data = 1; +} +``` + + + +**NS service**: + +```protobuf +service NS { + rpc Update(NameUpdate) returns (Response) {} + rpc Fetch(Query) returns (Content) {} +} + +message NameUpdate { + string name = 1; + bytes content = 2; +} + +message Query { + string name = 1; +} + +message Content { + bytes data = 1; +} + +message Response { + bytes data = 1; +} +``` + + + + +**Remote log:** + +```protobuf +message RemoteLog { + repeated Pair pair = 1; + bytes tail = 2; + + message Pair { + bytes remoteHash = 1; + bytes localHash = 2; + bytes data = 3; + } +} +``` + + + + + +## Synchronization + + + +### Roles + +There are four fundamental roles: + +1. Alice +2. Bob +2. Name system (NS) +3. Content-addressed storage (CAS) + +The *remote log* protobuf is what is stored at the Name system. + +"Bob" can represents anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS. + +### Flow + + + +

+ +
+ Figure 1: Remote log data synchronization. +

+ +### Remote log + +The remote log lets receiving nodes know what data they are missing. Depending +on the specific requirements and capabilities of the nodes and name system, the +information can be referred to differently. We distinguish between three rough +modes: + +1. Fully replicated log +2. Normal sized page with CAS mapping +3. "Linked list" mode - minimally sized page with CAS mapping + +**Data format:** + +``` +| H1_3 | H2_3 | +| H1_2 | H2_2 | +| H1_1 | H2_1 | +| ------------| +| next_page | +``` + +Here the upper section indicates a list of ordered pairs, and the lower section +contains the address for the next page chunk. `H1` is the native hash function, +and `H2` is the one used by the CAS. The numbers corresponds to the messages. + +To indicate which CAS is used, a remote log SHOULD use a multiaddr. + +**Embedded data:** + +A remote log MAY also choose to embed the wire payloads that corresponds to the +native hash. This bypasses the need for a dedicated CAS and additional +round-trips, with a trade-off in bandwidth usage. + +``` +| H1_3 | | C_3 | +| H1_2 | | C_2 | +| H1_1 | | C_1 | +| -------------| +| next_page | +``` + +Here `C` stands for the content that would be stored at the CAS. + +Both patterns can be used in parallel, e,g. by storing the last `k` messages +directly and use CAS pointers for the rest. Together with the `next_page` page +semantics, this gives users flexibility in terms of bandwidth and +latency/indirection, all the way from a simple linked list to a fully replicated +log. The latter is useful for things like backups on durable storage. + +### Next page semantics + +The pointer to the 'next page' is another remote log entry, at a previous point +in time. + + + +### Interaction with MVDS + +TBD. + + + +## Footnotes + +1. + +## Acknowledgements + +TBD. From c4183b25dc169f3acb908e48c8e2a2d65e94ce5c Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Tue, 1 Oct 2019 11:34:24 +0800 Subject: [PATCH 02/10] fixed review comments --- remote-log.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/remote-log.md b/remote-log.md index 9f0f0f4d..cefff4cb 100644 --- a/remote-log.md +++ b/remote-log.md @@ -52,6 +52,8 @@ Payloads are implemented using [protocol buffers v3](https://developers.google.c **CAS service**: ```protobuf +syntax = "proto3"; + package vac.cas; service CAS { @@ -73,6 +75,10 @@ message Content { **NS service**: ```protobuf +syntax = "proto3"; + +package vac.cas; + service NS { rpc Update(NameUpdate) returns (Response) {} rpc Fetch(Query) returns (Content) {} @@ -102,6 +108,10 @@ message Response { **Remote log:** ```protobuf +syntax = "proto3"; + +package vac.cas; + message RemoteLog { repeated Pair pair = 1; bytes tail = 2; @@ -133,7 +143,7 @@ There are four fundamental roles: The *remote log* protobuf is what is stored at the Name system. -"Bob" can represents anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS. +"Bob" can represent anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS. ### Flow From cef6fcc0b5e92ad9d15a25a1f33a432b8c9a2cc9 Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Tue, 1 Oct 2019 12:52:02 +0800 Subject: [PATCH 03/10] Update remote log asset --- assets/remotelog/remote-log.msc | 21 +++++++++++++++++++++ assets/remotelog/remote-log.png | Bin 0 -> 17444 bytes remote-log.md | 2 +- 3 files changed, 22 insertions(+), 1 deletion(-) create mode 100644 assets/remotelog/remote-log.msc create mode 100644 assets/remotelog/remote-log.png diff --git a/assets/remotelog/remote-log.msc b/assets/remotelog/remote-log.msc new file mode 100644 index 00000000..3e1baff4 --- /dev/null +++ b/assets/remotelog/remote-log.msc @@ -0,0 +1,21 @@ +# Alice and Bob: remote log data sync +msc { + hscale="2", wordwraparcs=on; + + alice [label="Alice"], + cas [label="CAS"], + ns [label="NS"], + bob [label="Bob"]; + + --- [label="Alice replicates data to a remote log"]; + alice => cas [label="Add content"]; + cas => alice [label="Address"]; + alice => ns [label="Update NameUpdate"]; + ns => alice [label="Response"]; + + --- [label="Bob comes online"]; + bob => ns [label="Fetch"]; + ns => bob [label="Content"]; + bob => cas [label="Fetch Query"]; + cas => bob [label="Content"]; +} diff --git a/assets/remotelog/remote-log.png b/assets/remotelog/remote-log.png new file mode 100644 index 0000000000000000000000000000000000000000..89266024299929f217bc099d4fd375a81fd9a73c GIT binary patch literal 17444 zcmch92{@H|zxL87DH@5;Dj^jTDRV0+4Me8QnKKqb=AtA!EEkv+DChKYtmBCU}*E~P*sEqg^G zQAtuS!QbrYUeAFqt1cctXH6ol%_n|Q`R!iIOd@R~$w(bmv=8cQaz3M|5;D0^S@M3{ zcF$#}W4Wk(wpF%1+a~_<+4du4GOyhCM9AK~_x_qf)kEQjy)o@fx_esrGNm?z8*AK5 zit>`Vx4r%S=5;IN$yBvbhp!#yTG#3}x$K!v@7&xa*CPKRcJ7P)0~5Uo)^eea1D(y| zU+2252afKP@xg(SNUCh96;zAAdw7^aP5egrjt4#Q_2tG@Wa6vpuJsh+>njCz58|s% z#BpkE&Fbpv0|yQ;dA>+bKNo*xr0w>J2ySlf^%Maip;*P3!Q2I{az?6?HI0o~7EKv$ zZf;kXu5@s8W5rrC+Q?+&+zQ;a?9bf?NhB@NK8(4{8=I-F*jOiQO~exQ?z1?%WAsp zYaT<@1_DgMK|xH+%-*bG^}|Z)>J4peZBLKf9Cy`m6SSWkI)CQO{c_`bckdQ`{;Za1 zYp$iGrK+mhFc_}6rUIXG@xz@>`}Xa-a^=cOI@ZzA(V#=tFJ{lKbz@LIcP{-xcinHgZRca=OTbC|f%B%6_TX*-=RA=aF za){^^T~kwaRn>zRD%c~!m+onF9L#=g*^*=3`DG2OsA_z4Z0ygGkrO9R&a|>EaMjvu zea>%bIr#YT?8?F~_uogh9?wr;5->lB&;x9mjHpKQb?0zPz(Elug9;r3P`C zmapDC+mJiQ?7p^ZhhBk~=bbwa3v(0drc3Zzy40?}zWTzD!BJku~vf(Ay%2;oYp-p}P0o^@4OzYP_tL+VZ@+2uKDdWP{&6{l}2AW>J zeEIOuT*(7rs z=e{uhI@hnLiMGO^=#hb;;e3Zs?o_Y#0=FxD&QwR>!l>LrmF4i#M#K>SXx;kJ(`WHj87Y^b{(7J$PJM;@Vd{N$1C1jiR^jE30|L%zX^G>HHJXojG&2lK zp7iyZ9XnPKu-|B8)2I^XcfZk(%$E5dJ`}d^)B7kTC3W0KpFhBZ{`&RnKI~!+qsOT) z&CgDSAM+MIbf_uYQN=xv=@}Uri8$#$P?yZf%^f2lNA*gvu&_`q?!ci#>6&81>hG;huq@dM zK#JGQd{Ntb^5jV|F|lJ_TX(%49vWg4G*P^8Av)`jb;qf6KAk+b=(CZJ{QP#UKjd)B zV@OF~fayB@@bIwS$9r4$8=jJumOgp12oNG{ZpI&g>(QVjEFxl>T3=v(+$Y7V{j)e7 z%PVT2m6w;7(U+hLDr^i449v{CGgzKW%nrTJoh`~0b(vLd{!mcR`JPUqG0nVTxV3

GF8KioiQL3Q`MCL`tWT9=+f&W@zt-2+ zH#AJBxY@5`_ej_tUe35-L!~W;UGd|6BMfC;v$L~UWL47vuN@}_WPCZgZ8Go!MgF`+ zeu9F6XJ4F3w;$G(2qx*wuivoYwOL)o_~xllm+4)coSdSf8a5)wgM)+VSVbo#7Jf>Y zJ7&5rifNthH+mHI=4_;_X0D57$aP}_KE?y)^)@r({a*b5oD{XV zw>n>6oj-E)=+P4=K3`kD8W=I^kgb)_l4c&5o$Y+$#0fpU@X%0giQuI=^9L*%?YqjJ zuG+xyQY*W@v@{_lML26;AvZQQrkT}k_qcWYwmgs1sN_Op%&idhG_!!nNTbXXQ{%ZT zBDS~I()m4L@17p(b;xe6uD(1u^nJJTOZ(BT2kbJZPoKVh`!+z2hlj@@J0~V4W}$RB zpXtP4i*};!_0Z7J@bK2h`}7{|Rky_G(?Oh zh@^8^jg7m~q$S6hLOu1AA8W@u#WCt$U|`@|oj1-iY3wquEt=F)j2|e+`WbPwd4GEM z?%ntIH(h6k3tU`WzI^#|{`~n(n>L*YT^4dx;uIY*+m-J)QWI<-@5BRog1@-=Z6_}y z{$cmge@_k!STl(K!qA zGcH~+N~V0g8)P&!o7`8@If$hlc3BM=H*F-mwz2rVrCA^oo(lRe`8IrpPrc00j83yD?j5k(kq6!MyB&#H#?ZVy|!GlUqMk(K>ySIQt=sL#4&jqj`!91^J))p z`@~;McNpu*9L$-S?oBXV9Cn4CoU%cxASyUFAd1BjaV%@#=*qQQ4?2u?d5cd+RTf>w ziUxquvWnWzP7VXwE~qaQx7yafh_mhQ?d@gY)e<%P`fA+{={;ImZ*p>SD(y1{lW#I; zQ=d07n&^%U_1b#K>RaB`WTVpf=;(9lR_(9Ozt?h|8w2bYnG3Ip*8D0q%UK>1Tlq2} zAymS3-ZUZmz@;Dg3AyuX5efmVx@x3X((8-2T5b<}>B8!c(S7mqWx;?`{yduNDf)(n zCw?=^uE_Yz&dt4l|GqTnP_pZSOPq3`kI!})h2zIR6$k9cu*sPlZ#a5)^M%xUPzXEG zXyEcdOYQ=mBwRsN)hOlIn{;gR?f%*wot>q@BDEzY{U8`|Z&N&<$Law8C=`k;h{{M? z;kR$!!os$t52swHR5*LK-74}}u5JI5?CiNGf~I%w-o4#`yY_>NMvwi!TsGEv*oZfv-87&f{%APWtGZn)bH0E8#Zz z`wKZuUc_Ku!+hYOpWlZMAL{Dso3kBX2i~v0(d6s89_#pWb!S_lF9-xUps@f`Q)8nC zJ!UdXX*}=HA!@F>`-}(d7m8Wk>CY%jc)t~Dw%@_SBg^BxWk1LE?UbJFWKoCFC|_US zAV$ob-?M;vmZQ{(SAH6%H&*k2K~Qe-=FTeoF{HZ+*jyVgP`+N^^I->;C@U*}_S>>$ zi-m=SRl1;FcTd-$Lx)UKGoC%8#Fu{g5+4&|Vz=3GyzkQG%h{L%Sy#Hs$YhTodhwrc zno=&rraI$F#KrZen%1^W$HZ`hWaHx7_B&P>$?@c{%cP~H%}fmX_xk$yI5|73)9v+m zW7Dl5L{1ee37PdZc9zTR*)Ji{^7id_P$sMFpl81HOO&tS3Yg8{!&&#$DYXw&HY&SRC;Q&7Z8&9=x7#X*P8#&5;suZlhkY9L};t;9#U1!m>TDaU&7 zV%VEElek$XmuHx_WC%|u85B{D9Jws4X>V_@s`~h?4x@nanM`M1&5Vs3H&&^!F)`Io zSM2w0cz+-c{QVD}Vd-FRe~(EpCMCsYdbFFjtaCqaj)twBo&Rbee@IA3Tze-?^vsdb zA%W8zzW)BTc)U1jS1wGUK2HmHJ;p&>|>QOOfQtPXHVQwF()(<@i5On<>y z&@U5cTW?@zlByP>>UdQ=^vbYfR{%LXk1X6@sn4GpH~6?qt(fuosQ{~jB)@xySARTW zK*}>?^!)hvF=MnR2(HcaOL?+KXlM(l8xs>#TM6W!+!Fzr$ja)Pnl$k_EKI}C4|f5% z`Z~gz27N+9c^RGr1_CU?E4v+7i!nrSvp_HayFwsmk#eEqvt8-w>4Vu5&0iDsT3i?A zz{ktV%I@8}XFCH?;WE?zI&P(yb;rG+wPS5>#4c10sQ#iXBZiBVMJY3OILq1Adb+eR z@g`Ypkety|310E+*^e*5B35&+TU;7^F%J`#Kmqg7Mal+exXd}2nVB`5aDRK>*Vp=Q zM22hEuBq9drDqispf;TUATkn{en?qZ#J|-sHZvRj`LnmR|+K_A1rq>vp%rx1s5F42-~5)gPgP^ z`%N-@)>*gRlBUgRdDPR2D=U*VBb*Xyy;ww~jO4atyUb;1CD>iQ%p&eId1upZRxt;5 z8Q||(UEODynVC2$t%SkJJ)32GJm?`^;{t;Rwo52>bd`sL>Ra2|hG3v!aXddjx9!5j z!*W^ox|9pP%CWdrkh0vncjMmshN`6-m9h#6z5ev+lf@EQEC`6Lix-DRM>R5S;|leA z?6*U4g4y6X!@G*u4~2o3f+zX5+E%}aOY666%GfFK?5wP_dg!YPWx46T!VmuJg*7z? zczNY`60~#0!G?KSAh)Q=xeS`us}A%LpfywVEPfiJ-B3!((On|NsL+qTKwKX-cPy{(Rq`SjNB(Zm59wavg zazeGDFiqC_6|{`^gM&5CocY$?9#Pv14yzoe7GZOSjbMs44|({z|E#C-WRV;=u| zHW#+xFV;?a>;d-`Yo#|fEwm)2SW_zWAc=hmBk=C=FO@k{7C!O zFABCWxEh@Ct`vpZX8ZpA`-O#t`S{9=1-My=zb;cbPJMOU1uXPuXX!D#q@)NH#FLx_ zA_T)_1yBlR#*SlNDQSZ-Il&nj89?~8Tln)E8XTK4F3W|8uA@}d*79m)*+V?+kiMsy zWq--pIlJ-emoEcD-`_(qrzIvr!A&-+k1h061=IJ9y~O@|7!F(yb$J64#kTp;nJr zO42$c9lnpBzvAP^j@sP$F=&K)(yi~Vug2A76LYW``t}y=c@??JBvmVcfnuZ7@+DEP z0Qwh78JzUtuCu2&&R)7S+_Eq~0lBH9tPK7KzW@vn$V_-40ENX72uEZ!87G9dAM35n z&dydB*@O4t;);rkt2P#(;11K;*-c!#cFnH&N@u|0A@UniU)2Wu?c(L7qoK*bZG`#< zS~`H4LX2u=*zCd$_%iK5|MclotEaKkwmced!YaGl+D=m^VqSxD`agQ);NbANE{y)& z`}caVk0K%>MmjuJZD_j|8*5symayy8)F8jW#PQm?bt_@>$jg`eIum+j%YKhM-cO5T z(NdkDE9wo}w3`XC0CT?`kknlRcbkmxbb#RdU{r%~6^W3O#XnJ~jpsV4^vG3o#10q3Yre3JzAm_vS07D7~Ha>fn zm6heiCeiHd0moH3{8D#C1mzZtkefGdeEjg?sI+v(n>PRk$jH%)+F7qgM{WF0_=SYT z+ib%v%9ezIU9s<5M3iFIZOImeXvN?f_%5)M`BqRF>|6X*!b0aq<;Jj)k@1tSA;!u6Svo<#L{hHM zRLai_4{%s5-w}MnM<#h#=MMnDZbsj}0BGfXP#fo?A62fY`yZ>+4y%~T1EghK7%kvk+&T+DY3GXx|i+^DtRr_a`Esj7mw+UY@< z_!lp}D%Ah{`ExzIBVZL6hR|4&Qmpca{QSPJiAR3#{+`v;WWhRQWetJP^7!!^urR1! z8gfR=N_?KfJLPT?$r|~=jvYI2;aui=wIRPg8Vdlc`1bGr37&>i-6~*wpPmw@+&kt# zmQoo@0xZl-KZ}Zr3JVKMOH&RnrzXRAhSR#5hGr`Z%W00Ycx+$4wv&`}17=`A zU8e^{39cbdQw*U?)-BLh05!lLrYQ~ig>0~i0Mj653myr`*h9S=C>}R%e7+vvW3Yka z)cw^41_ogCP;YU(0jFNvxpN2B_4OM!dTcU=hK7(zxWCa@xjKV((#kY{%qbnwdC0JHPUEOhN#NbNue-QuTk`iaI`Hx;K zGxIZp{tqAGv<>z3ssFr3*F4IW-~*PeSi#M5eWwUc5i<%<)1b^mdjo82Xy_trWZVhd zSXL3+Jv=;L%FE@c6OGHC$@VsY5n!x4WY6c{-T>LK!u@GtVtHw)9ChNQ){k&ix^A+$ zO#Zmb3xhztgr=8m_w z3W0MZlA+rct>oHn0q2HI&$B*AhU!MshVA*d4Zp(D^=_2$Ctll#@5(S^PhD6N!%`qQ-vPL!WnXb2IVle=-VnKA@XV?J2z< z9!kwW1GYY-z6kWNr5HRg2O&GV}a?SRwYmAIK!2n>c0=R*7gXv4`BjQ(J-sz;B z03rw5gOVaF=_n25CFls!*y2y0E?&F{C%qg--O4pg>>g)gs}UDi$x_nNH!?FF4KySPKbJi1I#PhnzbVES#E> zQq|lnP7qz!MWP$$2?LDqBSEr%{P+QHDW)Upzp1II-kP{Th!3z- z5it#ObMxL-RTC4F7ch{9e|)gm9t7$Nt3_A$2kuqdlaQ@Zfj|4X-Hf#EsC+jeeLewp z8m+i<=gt!=myjOa@Su0&hm$4I^yZQt2&pz46u8IYXb50JiwFqVAk4FJ-?UqKtGLsf zs3>Dx3N!@nczS9gA8~g-qNrE}o@cdw_H7ptJjLoj>rgO8r@3z&Dxa4!a}(00h# zkkx=)W!ZH5WSvAe0WKs83~w&Au4NUCs_aI@%`SsG{7+U1*oP1M1?UD41>AEVcF5M9 z92^-~PKe0H;M~S|k<&9X5sW9Bm*hClc+exsdLWc3d_OJ@#fXqb!@21(C=jbDh+7Kv zp?QlXT|Ej|eQqjJ1(v*-w*4kEG1n5QbZDQ5_~nNx9SW%wWgGKXlFMo{rn>iwi;IhiWlOkIkygK-U`#IfHN?UaF42*Pxb<)p1BAU{DDCpu z7+kC3D#@)5lofxogJ9>+CnxO*v9D&$W7SRK&{wQ%Nf9O^idu8&2p zSgEN!(XcQxgN4wEXcSn98pp#mv=}#s**!=vuVD>5m+@5-a(8!!^NYwrOPu_Wy?du4 zKKl&KNogw89Z1$8f>l&hewR^ge|_kmgQert^1n~29x>PcXUE04_v%_UaYPfwSg~wu zp$R$DF|l6^G?!yomN5{+G#I3xNZLRwK^-?yu%7D6CP7(5MW=5=*-c!#%d-vUV}w{Sfk`TH|_enhAT^Do`# z95jH5Ne^5i)LbBk3B?m^x9Q$`hE=$I%OBla@`?t!1OzxP5N7}|D~J4o{k9nPPuUwY-1%Q zrGhWl={1-C{30^Qb*Tgp-j3C&v#DN8v7^;n!t35W;@dDvPoBIR7^rS~(zj4iUOv&R zE~%{SOt35iEUV3%4fz-$Jcv7~JQwX5u_Twg#4|hwtsZ=3r^dL0QqC1i{R>ucu`lF#>VKIPel#p%T*NGO_ z+~?2P1q1|k>^NfK`?oj%xzg_4yFqo@reR7Vg9?_dtuD{($%@sP?qI_Df2Ciw9!lYP zWaMs^JlSxslR?z z1%nwJWMg5;zTB%!&If&%o0~H#eq3HsF4EsvP!L(EGmYs)BK79eK0W%N_Lk3@C1T%u zdqHos$oU8>j~qGD`s2s;9Xs%0=vzT(q88-$=n;rcT3%ircp)?G4g9xA1XIi!0X3X# zyTN7@Z~WOv*0YnBDyDWqU1s;p%tOuq!L{IvD?vRa8knuqm zXFVh1*9%LVb;Dk7+_dR5+7SBu51vyctevtx)6GTdB|`OLdbMjF9v<;YhXKRKgmANd z9;H~0Px8}{URgN>P;JC%q_;lh|C^LRNV!?!-jckcZ;tos;Ezb2WA|`NVss~db$B^a z@09}jvddsTDEy%u+&#*kfwG?T*~rG0u;|~Aib?yk3==9ZOsDNWeg*pGgTj`7C=%MI z2D;04;W9#h!Lym=Uiv?EvGKG2JL7~DHsQ%4qCDmbm$BLDXNjD|TuR_L3Mwctw7ff; z4;rV)tprET+B*3}#5AanpBE(4`B)o#V$nus=1j9XesVrIix;cdlgEz_0zTk&>|*gJ zx>WGQAhu(OOK=~PW*VGy1{mHpf~pb5IDX^=h0EhrV&YVh_WUy&2rBBgKJp~V+)H+L z48JqIolV_QuQ?rbP<~NI#Kvy&LtOID)xC<-p>XTa_UVQOB5DM9(w{teg8trG<^z3W zwc6;AWH!F|&(`>ZUg#Bmxba9kaB$P%-r_w1k1f0$QECC(hBr4kIq5W3tztf5lBu5( z`$+TXa+2gAQ5qS&%`9Yr*cLB>%E7ot>dwzd`_K420ys|_NCJaLX9s?th_Eo!&re7P zmdrJ)S7)NohrAl~AlR(v95yGNmhAlYynK0Tw&L%JJ4EKd%){Df8nT9lhNzgB zdeLIk9ue`Oq@)DY6)$A{C>%nNVX@G^iVCezlfec!*A6Q?9{{@t?lZ#_NS z8P$mfD)N_IOXAXqK>1wVVxDX`MbxSalZs%B3^Q__P3z#-dsS&A(%jq(K^Uc%?M1}qmylMb^H7K8Tf-woNd|S@$uU0vP!u`Pdu2AD0(pg^EboA;>l!NHfE>65t+iVZ=XP;1$c zRsuDP;A+t$ zRI6bM;0ItcLMqRB;e!!(L&)GDzz0!2M-~B&15&20qw@`;c+HCU_^-&o41a<1 z)hLHpH15@+U+Q0Yx!m-%1+%2YuZQ8+17Y~@eI;Uq{0j=-#e(GS%A_d>l2_F`<~si~bVi(q6uhWTBoolH z2^P5EQvm_83uG0Ufu*2;z#i?K#`o`!0M`SB24TTcZehi4-nbEwaB0B)V;1NMyuSGT z9U9j}V-y8s6eO(XN&*osX11VI{;(qOe+WpvO$B)6k7$B!F3(lKxBY;6sPVc;-_8);>24W+W!ZVmm1ryxhY?su8q zOPjt$u|Ohi-YzDl8KDqC@LKBnhFn)yG}i#T`ufjNOR%*~!vjRWor8nOEi^L}sn7p5 zhl^Oca!=@;>tmpbKTa=um=BkBr}taJY&KlJ$1kh%U#TDDn<3}`d=;G2!-f} zYpGt@I<0*oINt|#&1VT?y{S>?lZsN9+;I54^wyi^$pwAOewo9NhNj8F z#B#~SC#`cFF%PUA_DqE2)eGa*aK%Riuh_vMRhZ1oG+eACkw^(o!pK@c%>)d??gRToRV2o%3 z5p{kKc;0y^Ltv)?@y=u^1%-;)sh`*BQAPkq1c=}^#m)R{EBI%zGgTI85p=&zed?uH z;$Y%I^5!u~Fz1kXTOHu?K&_ zFv#J~U2s;rNB;x7>2qP>DmuD}v9UOq($UcmdSmg5iLnvZl*<6SN=;2gz4(z36!IU2 zs+Kof=}zeH2nD}~54l*V9oSZp@x~zy5WXy1w{o0%JXFXT3Ln#Dq>z(LCacBSV%rEt zEOv)@dwDIk#wsHj_NabaY{cKQa5x)V4Nn7`JAVE?S4-fW86V&EEE}>GFdGpr$-6l2 z@~oM0({KwRg6;ag4v#ljMn*)y{bAU=c`YNKB=x1=sN5FyQ>X5g8x!=H{`X`+tdYTa zub5cyrstb)MCLycV;x)lr)C@#^gB^~!5Qp9Qj&y@)vysK&YUFr8b?NWg5h~Ze^r^W zvFWQxb2my4Cx0XiKt4;SEz74Bg(NbeV67z}hyMz1AGej&zPVYJH2jcQsie~Lwegxh z?8}!eBQ7!3;w5%tMOo9I{kBn|>iR{4k~CyC!X!^z5IthLkR_@R(#<^) ztEhM)G{;;EU6}3k7d6JlB1|RFZ*%~mFkI0j1-m=-nD0ESquhnLLG2`K(knA!Wrx+i z$;gm>A`mU%%_UR8Ha~O|`2$S0-1+HVt9GBAC+Fu{7Q_%Gv$N~N3*@a_KywmdbQ$x5 zC3&&8c834oB}>M&S5W(smi`E}qMm%X?w2c6QBmPR4~mDt0b6?9XDiTwh*F5gkgruT zqfq;uK*xBrM%{7_d&dFmv-)ykD3g>6Js1ZITTO|#-7ew6F?0(EL2 zbMVFKz(-G>IP`tpcclfx0)o-dz#!uJbI|J7m%F=hJ+YI_p`$neAtd(Wz|?##_5*4x zMBEvb+Ik)z?YSNm=v@XLjV|$TvU80J*f{m4)(WrKZ&DRiBP)z6Rdw2o=mR*dCS)?&bGYZk zMkzYL?cL3L9muIDkP2(gEKEy7=g^V-rL9RC#+~H%aoITqL5KblU&iXZ26@I(lG;(H_*7DYQbGTIJyS$wzhK_ZA$dE|ucDB;1ph2nP-*Q2@u6F@#uL`-y?Zv7 zE){QyM)t&Y7zF5RlT3TNxMZsLt`r&+X8lm^o> zTfZLjeTSuv?a84|BexqlcI-goOcy~K+|;;L=Y6>w)Vo@~Fjw2lKnYwJer=HY|;rD=|9@rtGlzWlP1qsHSSB-AIDgD_(2RG^Jj=g z^wAs+|IKZ6>m(RxHpgE%HK}Wzp?wK!<)~YdFpImn%6)zc4A;&-s~hwr2e#eEAym$7 z`~PjWlFmq5TmM8{3SF>b?bfdT{%ohI5!cyPHr^VgU&W1SS8WoX9Wg5;snA8ErJ+H$ z2(uiJfc0HcRJ3~aYQm0OJ-T_z7SM_>40qcM#+Q?%$ifm-bx$JLfgK@#Fu# zZmeUjI2&&`pvc`UB`I_7@VDi&qm^p8GGQBQjGCH6L#gQ%R zsj1cf+CzaJTO)e2qP*fOOIQ{sfGfD;dI~L@cvhtfhJ7wHNPJQf(!mFmTNvGNlRAKt zcorBa=s%Z)N@!C*@bS?VU_xPjpufK(grHu@v zm-N3r#<%u*GY&@6+1YsxDF?C^v_ad=&QK#}-iHZMQK4|fnux64R#&I$mHtQ34em8w z(C=`7Uq(fB73Pq1t`p)~()H)~xUG$iD6yGq8ci!?G-MC8xw{Em|45@O9Pq<6CaK;H z_t`JBfA$lrV~~u*qw3(0Q6V3eKsfQF5;NAiJ2A`4TI*PgE#r&n!dw6h>?$Uh4`C?w7$}B$B=8Fci zocI*JO=@x5stG(Wlyjkf7Q4FBqX>F|twf@9+7gZBn3x{e-hY~Mp$9K*7ZR#PgZ#jO z>8YtP{g!6&qj?pGBRPoH-ExO51mQfBmv?n3w#PBxaX>+b=sy6UMMa@LFQT0LJ4Y^S zrWxKKh9Ufm>-4{c;_mM`qzwO%kS{a}tKXm1En)SZUQX1z(2DvCyADMFM5u6`j~UZ3 zG+_ye*Q)r5dfH+i%wHb+Nn10c&b882F3l_YfgFh*Ig)c+V5oR2QM?5;ca5rRCGsVR zfBfn)Z9HOJ_vR+SlJl&;!Zu)+p`xNXvGSVMqRu9T9s5g3ZItQ%6}_I%9q*OIF_0wh r=ZCg*WYepFBH

- +
Figure 1: Remote log data synchronization.

From 5fcb995738cee10995f22699b18368a219c300e2 Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Tue, 1 Oct 2019 13:09:41 +0800 Subject: [PATCH 04/10] Add MVDS interact --- remote-log.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/remote-log.md b/remote-log.md index 1fe20b3a..07a5f03a 100644 --- a/remote-log.md +++ b/remote-log.md @@ -1,6 +1,6 @@ # Remote log specification -> Version: 0.1.0 (Draft) +> Version: 0.1.1 (Draft) > > Authors: Oskar Thorén oskar@status.im, Dean Eigenmann dean@status.im @@ -213,9 +213,7 @@ in time. ### Interaction with MVDS -TBD. - - +`vac.mvds.Message` payloads are the only payloads that MUST be uploaded. Other messages types MAY be uploaded, depending on the implementation. ## Footnotes From 5740e95234e7ecf853d7236514a173af8a940104 Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Wed, 2 Oct 2019 14:07:31 +0800 Subject: [PATCH 05/10] review tweaks --- remote-log.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/remote-log.md b/remote-log.md index 07a5f03a..b680b64f 100644 --- a/remote-log.md +++ b/remote-log.md @@ -38,7 +38,7 @@ This specification is complemented by a proof of concept implementation 1 Date: Wed, 2 Oct 2019 14:08:25 +0800 Subject: [PATCH 06/10] fix link --- remote-log.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/remote-log.md b/remote-log.md index b680b64f..ab671db3 100644 --- a/remote-log.md +++ b/remote-log.md @@ -213,7 +213,7 @@ in time. ### Interaction with MVDS -[`vac.mvds.Message`](./mvds.md#payloads) payloads are the only payloads that MUST be uploaded. Other messages types MAY be uploaded, depending on the implementation. +[vac.mvds.Message](./mvds.md#payloads) payloads are the only payloads that MUST be uploaded. Other messages types MAY be uploaded, depending on the implementation. ## Footnotes From cbe9f1c2e03b24bf287635ec71884e1b22d9850d Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Thu, 3 Oct 2019 11:08:30 +0800 Subject: [PATCH 07/10] update readme remote log --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index d1ea01fb..53d803cb 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,9 @@ These protocols define various components of the [vac](https://vac.dev) stack. - [mvds](./mvds.md) - Ensure reliable messaging between peers across an unreliable peer-to-peer (P2P) network where they may be unreachable or unresponsive. + - [remote log](./remote-log.md) - Ensures a node can read data from a node that iss offline by replicating a local log remotely. + + ## Style guide All specs follow [RFC-2119](https://tools.ietf.org/html/rfc2119). From 77ae86a9a8cadf3ca0b32a2ad1fd69fabfacc7dc Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Thu, 3 Oct 2019 11:10:58 +0800 Subject: [PATCH 08/10] add inline seq id idea todo --- remote-log.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/remote-log.md b/remote-log.md index ab671db3..55a400f3 100644 --- a/remote-log.md +++ b/remote-log.md @@ -204,6 +204,8 @@ semantics, this gives users flexibility in terms of bandwidth and latency/indirection, all the way from a simple linked list to a fully replicated log. The latter is useful for things like backups on durable storage. + + ### Next page semantics The pointer to the 'next page' is another remote log entry, at a previous point From 572f4e5633c801c6d244dc907f1d05d1834a9c5b Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Thu, 3 Oct 2019 11:12:50 +0800 Subject: [PATCH 09/10] more todos --- remote-log.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/remote-log.md b/remote-log.md index 55a400f3..c54bdf2d 100644 --- a/remote-log.md +++ b/remote-log.md @@ -126,12 +126,10 @@ message RemoteLog { - + ## Synchronization - - ### Roles There are four fundamental roles: @@ -145,6 +143,9 @@ The *remote log* protobuf is what is stored in the name system. "Bob" can represent anything from 0 to N participants. Unlike Alice, Bob only needs read-only access to NS and CAS. + + + ### Flow @@ -204,8 +205,6 @@ semantics, this gives users flexibility in terms of bandwidth and latency/indirection, all the way from a simple linked list to a fully replicated log. The latter is useful for things like backups on durable storage. - - ### Next page semantics The pointer to the 'next page' is another remote log entry, at a previous point @@ -213,6 +212,8 @@ in time. + + ### Interaction with MVDS [vac.mvds.Message](./mvds.md#payloads) payloads are the only payloads that MUST be uploaded. Other messages types MAY be uploaded, depending on the implementation. From 08be7b83a79859101546d0c01a19e99f897308bd Mon Sep 17 00:00:00 2001 From: Oskar Thoren Date: Thu, 3 Oct 2019 11:13:50 +0800 Subject: [PATCH 10/10] todo --- remote-log.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/remote-log.md b/remote-log.md index c54bdf2d..2a62f1b3 100644 --- a/remote-log.md +++ b/remote-log.md @@ -156,6 +156,8 @@ The *remote log* protobuf is what is stored in the name system. Figure 1: Remote log data synchronization.

+ + ### Remote log The remote log lets receiving nodes know what data they are missing. Depending