vacp2p
/
research
mirror of https://github.com/vacp2p/research.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
research/remote_log
History
Oskar Thoren 67f0032349
add ipfs feasability 		2019-10-03 13:56:30 +08:00
..
src add data field and 0.1 spec 2019-09-30 13:00:51 +08:00
tests Make getContent tests pass 2019-09-25 16:25:15 +08:00
.gitignore Better dir structure 2019-08-06 13:15:27 +08:00
Makefile Simplify remote log by removing body with direct pairs 2019-09-25 16:34:53 +08:00
README.md nim twirp service experiment 2019-09-03 10:25:03 +02:00
ipfs-feasability.md add ipfs feasability 2019-10-03 13:56:30 +08:00
remote-log.msc change sequence diagram order 2019-09-24 19:28:52 +08:00
remote-log.png change sequence diagram order 2019-09-24 19:28:52 +08:00
remote_log_spec.md add data field and 0.1 spec 2019-09-30 13:00:51 +08:00

README.md

Remote log mocking

Using socket server and client in Nim to mimick behavior of a remote log for data sync.

See https://notes.status.im/Rwh-18AdSgKAkhfwfBE-OA# for current sketch. More full spec to come.

Interfaces

A server represents a remote log. A client is an endpoint.

We need something to represent storage as well.

TODO: Break down roles into individual actors as scripts.

Node

Represents an endpoint, like a data sync node. Multiple actors.

Outside of using transport to send "live" messages (outside of scope here), it also uploads them to CAS and NS. Flow looks different for sending node and reciving node.

Sending node flow:

  1. CAS Update
  2. NS Update

Receiving node flow:

  1. NS fetch
  2. CAS fetch

Details to come.

Name system (NS)

E.g. DNS/ENS/Feeds. Fixed name for mutable resoures.

Supports which operations:

  • Update name with data => Ok/Error
  • Fetch name => data

Content-addressed storage (CAS)

Request/response interface for immutable data.

Supports two operations:

  • Upload data => id
  • Fetch id => data

Sketching interface

NS POST <> NS GET <>

CAS POST <> CAS GET <>

What's in <>?

  • hash string

  • blob

  • h1_3, h2_3], [h1_2, h2_2, next_page_chunk]

  • How do we want to encode this?

  • Or we can use protobuf or json

  • This seems mostly relevant for ns_head

  • lets just start with json thingy and NS/CAS sep

HASH1 and HASH2, all that matters. HASH1 is native for MVDS message (name?) and HASH2 native to the remote log CAS.

For NS, we want to know how to update it in/communicate it.

And then actual format with mapping and paging, in a way that's adjustable, and so on.

Can we generalize this to pack file for backups? Ie type and pointer to more etc

This would allow to go download whole thingy

also support many remote logs kind of thing, default or w/e

can we remove? scrub somehow.

Protobuf next

Actually use them, and check enum and dispatching etc.

Deps

Put in Makefile

nimble install https://github.com/PMunch/protobuf-nim

Protobuf service generation

Install nimtwirp, then: nimtwirp_build -I:. --out:. protocol.proto