From 5d3d477199a16eb5cb7d0b4b7abc2bb2d2ac1d4f Mon Sep 17 00:00:00 2001 From: David Dias Date: Fri, 23 Sep 2016 19:09:52 +0800 Subject: [PATCH 01/17] docs: add codecs for serialization formats --- multicodec-packed.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/multicodec-packed.md b/multicodec-packed.md index 07f6485..2fb8c54 100644 --- a/multicodec-packed.md +++ b/multicodec-packed.md @@ -71,7 +71,9 @@ code codec 0x42 multiaddr # Serialization formats (cbor, ion, protobuf, etc) -# TODO +0x50 protobuf +0x51 cbor +0x60 rlp # VCS'es formats (git, hg, SVN, etc) # TODO From 7d08aa42064e9f534263c44c97805221aa8604fd Mon Sep 17 00:00:00 2001 From: David Dias Date: Sun, 25 Sep 2016 15:46:07 +0800 Subject: [PATCH 02/17] fix: fix some mistakes in the spec --- README.md | 123 +++++++++++++++++++++++++++++------------------------- 1 file changed, 65 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index 40715e1..aa3ab8f 100644 --- a/README.md +++ b/README.md @@ -4,16 +4,16 @@ [![](https://img.shields.io/badge/project-multiformats-blue.svg?style=flat-square)](http://github.com/multiformats/multiformats) [![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs) -> self-describing codecs +> Make data and streams self-described by prefixing them with human readable or binary packed codecs. `multicodec` offers a base table, but can also be extended with extra tables by application basis. ## Table of Contents - [Motivation](#motivation) -- [How does it work? - Protocol Description](#how-does-it-work---protocol-description) -- [Prefix examples](#prefix-examples) -- [prefix - codec - desc](#prefix---codec---desc) -- [The protocol path](#the-protocol-path) + - [How does it work? - Protocol Description](#how-does-it-work---protocol-description) + - [The protocol path](#the-protocol-path) +- [Multicodec table](#multicodec-table) - [Implementations](#implementations) + - [Implementation details]() - [FAQ](#faq) - [Maintainers](#maintainers) - [Contribute](#contribute) @@ -38,67 +38,43 @@ Moreover, this self-description allows straightforward layering of protocols wit `multicodec` is a _self-describing multiformat_, it wraps other formats with a tiny bit of self-description: ```sh - -# or -\n + ``` For example, let's encode a json doc: -```node -> // encode some json -> var str = JSON.stringify({"hello":"world"}) -> var buf = multicodec.encode('json', str) // prepends multistream.header('/json') -> buf - -> buf.toString('hex') -062f6a736f6e2f7b2268656c6c6f223a22776f726c64227d -> // decode, and find out what is in buf -> multicodec.decode(buf) -{ "codec": "json", "data": '{"hello": "world"}' } +```JavaScript +// encode some json +const buf = new Buffer(JSON.stringify({ hello: 'world' })) + +const prefixedBuf = multicodec.addPrefix('json', str) // prepends multicodec ('json') +console.log(prefixedBuf) +// + +const.log(prefixedBuf.toString('hex')) +// 062f6a736f6e2f7b2268656c6c6f223a22776f726c64227d + +// let's get the Multicodec, and get the data back + +const codec = multicodec.getMulticodec(prefixedBuf) +console.log(codec) +// json + +console.log(multicodec.rmPrefix(prefixedBuf).toString()) +// "{ \"hello\": \"world\" } ``` So, `buf` is: ``` hex: 062f6a736f6e2f7b2268656c6c6f223a22776f726c64227d -ascii: json/\n{"hello":"world"} +ascii: /json/"{\"hello\":\"world\"} ``` -The more you know! Let's try it again, this time with protobuf: +Note that on the ascii version, the varint at the beginning is not being represented, you should account that. -``` -cat proto.c -``` +For a binary packed version of the multicodecs, see [multicodec-packed](./multicodec-packed.md). -See also: [multicodec-packed](./multicodec-packed.md). - -## Prefix examples - - -| prefix | codec | desc | type | [packed encoding](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md)| -|----------------|-------|-------------|-------|---------------------------------------| -|0x052f62696e2f | /bin/ |raw binary |binary | 0x00 | -|0x042f62322f | /b2/ |ascii base2 |binary | | -|0x052f6231362f | /b16/ |ascii base16 |hex | | -|0x052f6233322f | /b32/ |ascii base32 | | | -|0x052f6235382f | /b58/ |ascii base58 | | | -|0x052f6236342f | /b64/ |ascii base64 | | | -|0x062f6a736f6e2f |/json/ | |json | | -|0x062f63626f722f |/cbor/ | |json | | -|0x062f62736f6e2f |/bson/ | |json | | -|0x072f626a736f6e2f|/bjson/| |json | | -|0x082f75626a736f6e2f| /ubjson/| |json | | -|0x182f6d756c7469636f6465632f | /multicodec/ | | multiformat | 0x40 | -|0x162f6d756c7469686173682f | /multihash/ | | multiformat | 0x41 | -|0x162f6d756c7469616464722f | /multiaddr/ | | multiformat | 0x42 | -|0x0a2f70726f746f6275662f |/protobuf/ | Protocol Buffers |protobuf| | -|0x072f6361706e702f | /capnp/ | Cap-n-Proto |protobuf| | -|0x092f666c61746275662f |/flatbuf/ | FlatBuffers |protobuf| | -|0x052f7461722f |/tar/ | | archive | | -|0x052f7a69702f |/zip/ | | archive | | -|0x052f706e672f | /png/ | | archive | | -|0x052f726c702f | /rlp/ | recursive length prefix | ethereum | 0x60 | ## The protocol path `multicodec` allows us to specify different protocols in a universal namespace, that way being able to recognize, multiplex, and embed them easily. We use the notion of a `path` instead of an `id` because it is meant to be a Unix-friendly URI. @@ -115,17 +91,48 @@ An example of a _great_ path name is: ``` /ipfs/Qmaa4Rw81a3a1VEx4LxB7HADUAXvZFhCoRdBzsMZyZmqHD/ipfs.protocol -/http/w3id.org/ipfs/ipfs-1.1.0.json +/http/w3id.org/ipfs/1.1.0 ``` -These path names happen to be resolvable -- not just in a "multicodec muxer(e.g [multistream]())" but -- in the internet as a whole (provided the program (or OS) knows how to use the `/ipfs` and `/http` protocols). +These path names happen to be resolvable -- not just in a "multicodec muxer(e.g [multistream](https://github.com/multiformats/multistream))" but -- in the internet as a whole (provided the program (or OS) knows how to use the `/ipfs` and `/http` protocols). + +## Multicodec table + +| prefix | codec | desc | type | [packed encoding](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md)| +|----------------|-------|--------------|--------|---------------------------------------| +|0x052f62696e2f | /bin/ | raw binary | binary | 0x00 | +|0x042f62322f | /b2/ | ascii base2 | binary | | +|0x052f6231362f | /b16/ | ascii base16 | hex | | +|0x052f6233322f | /b32/ | ascii base32 | | | +|0x052f6235382f | /b58/ | ascii base58 | | | +|0x052f6236342f | /b64/ | ascii base64 | | | +|0x062f6a736f6e2f |/json/ | |json | | +|0x062f63626f722f |/cbor/ | |json | | +|0x062f62736f6e2f |/bson/ | |json | | +|0x072f626a736f6e2f|/bjson/| |json | | +|0x082f75626a736f6e2f| /ubjson/| |json | | +|0x182f6d756c7469636f6465632f | /multicodec/ | | multiformat | 0x40 | +|0x162f6d756c7469686173682f | /multihash/ | | multiformat | 0x41 | +|0x162f6d756c7469616464722f | /multiaddr/ | | multiformat | 0x42 | +|0x0a2f70726f746f6275662f |/protobuf/ | Protocol Buffers |protobuf| | +|0x072f6361706e702f | /capnp/ | Cap-n-Proto |protobuf| | +|0x092f666c61746275662f |/flatbuf/ | FlatBuffers |protobuf| | +|0x052f7461722f |/tar/ | | archive | | +|0x052f7a69702f |/zip/ | | archive | | +|0x052f706e672f | /png/ | | archive | | +|0x052f726c702f | /rlp/ | recursive length prefix | ethereum | 0x60 | ## Implementations -- [go-multicodec](https://github.com/multiformats/go-multicodec) -- [go-multistream](https://github.com/multiformats/go-multistream) - Implements multistream, which uses multicodec for stream negotiation -- [js-multistream](https://github.com/multiformats/js-multistream-select) - Implements multistream, which uses multicodec for stream negotiation -- [clj-multicodec](https://github.com/multiformats/clj-multicodec) +- multicodec + - [go-multicodec](https://github.com/multiformats/go-multicodec) +- multicodec-packed + - [go-multicodec-packed](https://github.com/multiformats/go-multicodec-packed) + - [js-multicodec-packed](https://github.com/multiformats/js-multicodec-packed) +- multistream + - [go-multistream](https://github.com/multiformats/go-multistream) - Implements multistream, which uses multicodec for stream negotiation + - [js-multistream](https://github.com/multiformats/js-multistream) - Implements multistream, which uses multicodec for stream negotiation + - [clj-multicodec](https://github.com/greglook/clj-multicodec) ## FAQ From 21605069a9396fc59963ece9034d3107d25341d7 Mon Sep 17 00:00:00 2001 From: David Dias Date: Sun, 25 Sep 2016 16:19:10 +0800 Subject: [PATCH 03/17] update table --- README.md | 72 +++++++++++++++++++++++++++++--------------- multicodec-packed.md | 32 -------------------- 2 files changed, 48 insertions(+), 56 deletions(-) diff --git a/README.md b/README.md index aa3ab8f..b499663 100644 --- a/README.md +++ b/README.md @@ -98,29 +98,53 @@ These path names happen to be resolvable -- not just in a "multicodec muxer(e.g ## Multicodec table -| prefix | codec | desc | type | [packed encoding](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md)| -|----------------|-------|--------------|--------|---------------------------------------| -|0x052f62696e2f | /bin/ | raw binary | binary | 0x00 | -|0x042f62322f | /b2/ | ascii base2 | binary | | -|0x052f6231362f | /b16/ | ascii base16 | hex | | -|0x052f6233322f | /b32/ | ascii base32 | | | -|0x052f6235382f | /b58/ | ascii base58 | | | -|0x052f6236342f | /b64/ | ascii base64 | | | -|0x062f6a736f6e2f |/json/ | |json | | -|0x062f63626f722f |/cbor/ | |json | | -|0x062f62736f6e2f |/bson/ | |json | | -|0x072f626a736f6e2f|/bjson/| |json | | -|0x082f75626a736f6e2f| /ubjson/| |json | | -|0x182f6d756c7469636f6465632f | /multicodec/ | | multiformat | 0x40 | -|0x162f6d756c7469686173682f | /multihash/ | | multiformat | 0x41 | -|0x162f6d756c7469616464722f | /multiaddr/ | | multiformat | 0x42 | -|0x0a2f70726f746f6275662f |/protobuf/ | Protocol Buffers |protobuf| | -|0x072f6361706e702f | /capnp/ | Cap-n-Proto |protobuf| | -|0x092f666c61746275662f |/flatbuf/ | FlatBuffers |protobuf| | -|0x052f7461722f |/tar/ | | archive | | -|0x052f7a69702f |/zip/ | | archive | | -|0x052f706e672f | /png/ | | archive | | -|0x052f726c702f | /rlp/ | recursive length prefix | ethereum | 0x60 | +| prefix | codec | description | [packed](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md)| +|-------------------------------|---------------|-------------------------|-----------| +| **Miscelaneous** | +| 0x052f62696e2f | /bin/ | raw binary | 0x00 | +| **Bases encodings** | +| 0x042f62322f | /b2/ | ascii base2 | | +| 0x052f6231362f | /b16/ | ascii base16 | | +| 0x052f6233322f | /b32/ | ascii base32 | | +| 0x052f6235382f | /b58/ | ascii base58 | | +| 0x052f6236342f | /b64/ | ascii base64 | | +| **Serialization formats** | +| 0x062f6a736f6e2f | /json/ | | | +| 0x062f63626f722f | /cbor/ | | | +| 0x062f62736f6e2f | /bson/ | | | +| 0x072f626a736f6e2f | /bjson/ | | | +| 0x082f75626a736f6e2f | /ubjson/ | | | +| 0x0a2f70726f746f6275662f | /protobuf/ | Protocol Buffers | | +| 0x072f6361706e702f | /capnp/ | Cap-n-Proto | | +| 0x092f666c61746275662f | /flatbuf/ | FlatBuffers | | +| 0x052f726c702f | /rlp/ | recursive length prefix | 0x60 | +| **Multiformats** | +| 0x182f6d756c7469636f6465632f | /multicodec/ | | 0x40 | +| 0x162f6d756c7469686173682f | /multihash/ | | 0x41 | +| 0x162f6d756c7469616464722f | /multiaddr/ | | 0x42 | +| **Multihashes** | +| | | | | +| **Multiaddrs** | +| | | | | +| **Archiving formats** | +| 0x052f7461722f | /tar/ | | | +| 0x052f7a69702f | /zip/ | | | +| **Image formats** | +| 0x052f706e672f | /png/ | | | +| | /jpg/ | | | +| **Video formats** | +| | /mp4/ | | | +| | /mkv/ | | | +| **Blockchain formats** | +| n/a | n/a | n/a | n/a | +| **VCS formats** | +| n/a | n/a | n/a | n/a | +| **IPLD formats** | +| | /dag-pb/ | MerkleDAG protobuf | | +| | /dag-cbor/ | MerkleDAG cbor | | +| | /eth-rlp/ | Ethereum Block RLP | | + + ## Implementations @@ -134,7 +158,6 @@ These path names happen to be resolvable -- not just in a "multicodec muxer(e.g - [js-multistream](https://github.com/multiformats/js-multistream) - Implements multistream, which uses multicodec for stream negotiation - [clj-multicodec](https://github.com/greglook/clj-multicodec) - ## FAQ > **Q. Why?** @@ -172,3 +195,4 @@ Check out our [contributing document](https://github.com/multiformats/multiforma ## License [MIT](LICENSE) + diff --git a/multicodec-packed.md b/multicodec-packed.md index 2fb8c54..f338872 100644 --- a/multicodec-packed.md +++ b/multicodec-packed.md @@ -50,38 +50,6 @@ It is worth noting that multicodec-packed works very well in conjunction with [m Multicodec-packed uses "protocol tables" to agree upon the mapping from one multicodec-packed code (a single varint). These tables map an `` to a full [multicodec protocol path](./README.md#the-protocol-path). These tables can be application specific, though -- like [with](https://github.com/multiformats/multihash) [other](https://github.com/multiformats/multibase) [multiformats](https://github.com/multiformats/multiaddr) -- we will keep a globally agreed upon table with common protocols and formats. -### Standard mcp protocol table - -This is the standard multicodec-packed protocol table. - -**WARNING: WIP. this table is not ready for wide use.** - -TODO: -- [ ] See if IANA has a ready-made table for us to use here. Even just a listing of the most popular formats would be good enough. - -```sh -code codec - -# Miscellaneous -0x00 raw binary data - -# Multiformats -0x40 multicodec -0x41 multihash -0x42 multiaddr - -# Serialization formats (cbor, ion, protobuf, etc) -0x50 protobuf -0x51 cbor -0x60 rlp - -# VCS'es formats (git, hg, SVN, etc) -# TODO - -# Blockchain block types (bitcoin, ethereum, stellar, etc) -# TODO -``` - ## Implementations - [go](https://github.com/multiformats/go-multicodec-packed/) From d8c45d03cf7df2688664c9925d16115beb70c483 Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 26 Sep 2016 11:47:37 +0800 Subject: [PATCH 04/17] add multihash, multiaddrs and multibase tables --- README.md | 122 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 75 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index b499663..fc80f02 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ Moreover, this self-description allows straightforward layering of protocols wit `multicodec` is a _self-describing multiformat_, it wraps other formats with a tiny bit of self-description: ```sh - +\n ``` For example, let's encode a json doc: @@ -98,53 +98,80 @@ These path names happen to be resolvable -- not just in a "multicodec muxer(e.g ## Multicodec table -| prefix | codec | description | [packed](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md)| -|-------------------------------|---------------|-------------------------|-----------| -| **Miscelaneous** | -| 0x052f62696e2f | /bin/ | raw binary | 0x00 | -| **Bases encodings** | -| 0x042f62322f | /b2/ | ascii base2 | | -| 0x052f6231362f | /b16/ | ascii base16 | | -| 0x052f6233322f | /b32/ | ascii base32 | | -| 0x052f6235382f | /b58/ | ascii base58 | | -| 0x052f6236342f | /b64/ | ascii base64 | | -| **Serialization formats** | -| 0x062f6a736f6e2f | /json/ | | | -| 0x062f63626f722f | /cbor/ | | | -| 0x062f62736f6e2f | /bson/ | | | -| 0x072f626a736f6e2f | /bjson/ | | | -| 0x082f75626a736f6e2f | /ubjson/ | | | -| 0x0a2f70726f746f6275662f | /protobuf/ | Protocol Buffers | | -| 0x072f6361706e702f | /capnp/ | Cap-n-Proto | | -| 0x092f666c61746275662f | /flatbuf/ | FlatBuffers | | -| 0x052f726c702f | /rlp/ | recursive length prefix | 0x60 | -| **Multiformats** | -| 0x182f6d756c7469636f6465632f | /multicodec/ | | 0x40 | -| 0x162f6d756c7469686173682f | /multihash/ | | 0x41 | -| 0x162f6d756c7469616464722f | /multiaddr/ | | 0x42 | -| **Multihashes** | -| | | | | -| **Multiaddrs** | -| | | | | -| **Archiving formats** | -| 0x052f7461722f | /tar/ | | | -| 0x052f7a69702f | /zip/ | | | -| **Image formats** | -| 0x052f706e672f | /png/ | | | -| | /jpg/ | | | -| **Video formats** | -| | /mp4/ | | | -| | /mkv/ | | | -| **Blockchain formats** | -| n/a | n/a | n/a | n/a | -| **VCS formats** | -| n/a | n/a | n/a | n/a | -| **IPLD formats** | -| | /dag-pb/ | MerkleDAG protobuf | | -| | /dag-cbor/ | MerkleDAG cbor | | -| | /eth-rlp/ | Ethereum Block RLP | | - +| prefix | codec | description | [packed][p] | multibase only | +|-------------------------------|-----------------|-------------------------|-------------|----------------| +| **Miscelaneous** | +| 0x052f62696e2f | /bin/ | raw binary | 0x00 | n/a | +| **Bases encodings** | +| | /base1/ | unary | | 1 | +| | /base2/ | binary (0 and 1) | | 0 | +| | /base8/ | octal | | 7 | +| | /base10/ | decimal | | 9 | +| | /base16/ | hexadecimal | | F, f | +| | /base32/ | rfc4648 | | U, u | +| | /base32hex/ | rfc4648 | | V, v | +| | /base58flickr/ | base58 flicker | | Z | +| | /base58btc/ | base58 bitcoin | | z | +| | /base64/ | rfc4648 | | y | +| | /base64url/ | rfc4648 | | Y | +| **Serialization formats** | +| 0x062f6a736f6e2f | /json/ | | | n/a | +| 0x062f63626f722f | /cbor/ | | | n/a | +| 0x062f62736f6e2f | /bson/ | | | n/a | +| 0x072f626a736f6e2f | /bjson/ | | | n/a | +| 0x082f75626a736f6e2f | /ubjson/ | | | n/a | +| 0x0a2f70726f746f6275662f | /protobuf/ | Protocol Buffers | | n/a | +| 0x072f6361706e702f | /capnp/ | Cap-n-Proto | | n/a | +| 0x092f666c61746275662f | /flatbuf/ | FlatBuffers | | n/a | +| 0x052f726c702f | /rlp/ | recursive length prefix | 0x60 | n/a | +| **Multiformats** | +| 0x182f6d756c7469636f6465632f | /multicodec/ | | 0x30 | n/a | +| 0x162f6d756c7469686173682f | /multihash/ | | 0x31 | n/a | +| 0x162f6d756c7469616464722f | /multiaddr/ | | 0x32 | n/a | +| **Multihashes** | +| | /sha1/ | | 0x11 | n/a | +| | /sha2-256/ | | 0x12 | n/a | +| | /sha2-512/ | | 0x13 | n/a | +| | /sha3-256/ | | 0x16 | n/a | +| | /sha3-384/ | | 0x15 | n/a | +| | /sha3-512/ | | 0x14 | n/a | +| | /shake-128/ | | 0x18 | n/a | +| | /shake-256/ | | 0x19 | n/a | +| | /blake2b/ | | 0x40 | n/a | +| | /blake2s/ | | 0x41 | n/a | +| **Multiaddrs** | +| | /ip4/ | | 0x04 | n/a | +| | /ip6/ | | 0x29 | n/a | +| | /tcp/ | | 0x06 | n/a | +| | /udp/ | | 0x11 | n/a | +| | /dccp/ | | 0x21 | n/a | +| | /sctp/ | | 0x84 | n/a | +| | /udt/ | | 0x012D | n/a | +| | /utp/ | | 0x012E | n/a | +| | /ipfs/ | | 0x2A | n/a | +| | /http/ | | 0x01E0 | n/a | +| | /https/ | | 0x01BB | n/a | +| | /ws/ | | 0x01DD | n/a | +| | /onion/ | | 0x01BC | n/a | +| **Archiving formats** | +| 0x052f7461722f | /tar/ | | | n/a | +| 0x052f7a69702f | /zip/ | | | n/a | +| **Image formats** | +| 0x052f706e672f | /png/ | | | n/a | +| | /jpg/ | | | n/a | +| **Video formats** | +| | /mp4/ | | | n/a | +| | /mkv/ | | | n/a | +| **Blockchain formats** | +| | | | | n/a | +| **VCS formats** | +| | | | | n/a | +| **IPLD formats** | +| | /dag-pb/ | MerkleDAG protobuf | | n/a | +| | /dag-cbor/ | MerkleDAG cbor | | n/a | +| | /eth-rlp/ | Ethereum Block RLP | | n/a | +Note: Base encodings present a multibase encoding special char, so that the codec doesn't break the encoded version of the data it is describing. ## Implementations @@ -196,3 +223,4 @@ Check out our [contributing document](https://github.com/multiformats/multiforma [MIT](LICENSE) +[p](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md) From b923744653abb9756b963d2cea086e2dc9ed89a2 Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 26 Sep 2016 16:06:15 +0800 Subject: [PATCH 05/17] update with regards to https://github.com/multiformats/multicodec/pull/16\#issuecomment-249497577 --- README.md | 282 ++++++++++++++++++------------------------- multicodec-packed.md | 93 -------------- 2 files changed, 118 insertions(+), 257 deletions(-) delete mode 100644 multicodec-packed.md diff --git a/README.md b/README.md index fc80f02..692c256 100644 --- a/README.md +++ b/README.md @@ -4,16 +4,15 @@ [![](https://img.shields.io/badge/project-multiformats-blue.svg?style=flat-square)](http://github.com/multiformats/multiformats) [![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs) -> Make data and streams self-described by prefixing them with human readable or binary packed codecs. `multicodec` offers a base table, but can also be extended with extra tables by application basis. +> compact self-describing codecs. Save space by using predefined multicodec tables. ## Table of Contents - [Motivation](#motivation) - - [How does it work? - Protocol Description](#how-does-it-work---protocol-description) - - [The protocol path](#the-protocol-path) -- [Multicodec table](#multicodec-table) +- [How does it work? - Protocol Description](#how-does-it-work---protocol-description) +- [Multicodec tables](#multicodec-tables) + - [Standard multicodec table](#standard-mcp-protocol-table) - [Implementations](#implementations) - - [Implementation details]() - [FAQ](#faq) - [Maintainers](#maintainers) - [Contribute](#contribute) @@ -21,193 +20,150 @@ ## Motivation -Multicodecs are self-describing protocol/encoding streams. (Note that a file is a stream). It's designed to address the perennial problem: +[Multistreams](https://github.com/multiformats/multistream) are self-describing protocol/encoding streams. Multicodec uses an agreed-upon "protocol table". It is designed for use in short strings, such as keys or identifiers (i.e [CID](https://github.com/ipld/cid)). -> I have a bitstring, what codec is the data coded with!? +## Protocol Description - How does the protocol work? -Instead of arguing about which data serialization library is the best, let's just pick the simplest one now, and build _upgradability_ into the system. Choices are never _forever_. Eventually all systems are changed. So, embrace this fact of reality, and build change into your system now. +`multicodec` is a _self-describing multiformat_, it wraps other formats with a tiny bit of self-description. A multicodec identifier is both a varint and the code identifying the following data, this means that the most significant bit of every multicodec code is reserved to signal the continuation. -Multicodec frees you from the tyranny of past mistakes. Instead of trying to figure it all out beforehand, or continue using something that we can all agree no longer fits, why not allow the system to _evolve_ and _grow_ with the use cases of today, not yesterday. - -To decode an incoming stream of data, a program must either (a) know the format of the data a priori, or (b) learn the format from the data itself. (a) precludes running protocols that may provide one of many kinds of formats without prior agreement on which. multistream makes (b) neat using self-description. - -Moreover, this self-description allows straightforward layering of protocols without having to implement support in the parent (or encapsulating) one. - -## How does it work? - Protocol Description - -`multicodec` is a _self-describing multiformat_, it wraps other formats with a tiny bit of self-description: +This way, a chunk of data identified by multicodec will look like this: ```sh -\n + +# To reduce the cognitive load, we sometimes might write the same line as: + ``` -For example, let's encode a json doc: - -```JavaScript -// encode some json -const buf = new Buffer(JSON.stringify({ hello: 'world' })) - -const prefixedBuf = multicodec.addPrefix('json', str) // prepends multicodec ('json') -console.log(prefixedBuf) -// - -const.log(prefixedBuf.toString('hex')) -// 062f6a736f6e2f7b2268656c6c6f223a22776f726c64227d - -// let's get the Multicodec, and get the data back - -const codec = multicodec.getMulticodec(prefixedBuf) -console.log(codec) -// json - -console.log(multicodec.rmPrefix(prefixedBuf).toString()) -// "{ \"hello\": \"world\" } -``` - -So, `buf` is: +Another useful scenario is when using the multicodec-packed as part of the keys to access data, example: ``` -hex: 062f6a736f6e2f7b2268656c6c6f223a22776f726c64227d -ascii: /json/"{\"hello\":\"world\"} +# suppose we have a value and a key to retrieve it +"" -> + +# we can use multicodec-packed with the key to know what codec the value is in +"" -> ``` -Note that on the ascii version, the varint at the beginning is not being represented, you should account that. +It is worth noting that multicodec-packed works very well in conjunction with [multihash](https://github.com/multiformats/multihash) and [multiaddr](https://github.com/multiformats/multiaddr), as you can prefix those values with a multicodec-packed to tell what they are. -For a binary packed version of the multicodecs, see [multicodec-packed](./multicodec-packed.md). +## Multicodec-Packed Protocol Tables -## The protocol path - -`multicodec` allows us to specify different protocols in a universal namespace, that way being able to recognize, multiplex, and embed them easily. We use the notion of a `path` instead of an `id` because it is meant to be a Unix-friendly URI. - -A good path name should be decipherable -- meaning that if some machine or developer -- who has no idea about your protocol -- encounters the path string, they should be able to look it up and resolve how to use it. - -An example of a good path name is: - -``` -/bittorrent.org/1.0 -``` - -An example of a _great_ path name is: - -``` -/ipfs/Qmaa4Rw81a3a1VEx4LxB7HADUAXvZFhCoRdBzsMZyZmqHD/ipfs.protocol -/http/w3id.org/ipfs/1.1.0 -``` - -These path names happen to be resolvable -- not just in a "multicodec muxer(e.g [multistream](https://github.com/multiformats/multistream))" but -- in the internet as a whole (provided the program (or OS) knows how to use the `/ipfs` and `/http` protocols). +Multicodecuses "protocol tables" to agree upon the mapping from one multicodec code (a single varint). These tables can be application specific, though -- like [with](https://github.com/multiformats/multihash) [other](https://github.com/multiformats/multibase) [multiformats](https://github.com/multiformats/multiaddr) -- we will keep a globally agreed upon table with common protocols and formats. ## Multicodec table -| prefix | codec | description | [packed][p] | multibase only | -|-------------------------------|-----------------|-------------------------|-------------|----------------| -| **Miscelaneous** | -| 0x052f62696e2f | /bin/ | raw binary | 0x00 | n/a | -| **Bases encodings** | -| | /base1/ | unary | | 1 | -| | /base2/ | binary (0 and 1) | | 0 | -| | /base8/ | octal | | 7 | -| | /base10/ | decimal | | 9 | -| | /base16/ | hexadecimal | | F, f | -| | /base32/ | rfc4648 | | U, u | -| | /base32hex/ | rfc4648 | | V, v | -| | /base58flickr/ | base58 flicker | | Z | -| | /base58btc/ | base58 bitcoin | | z | -| | /base64/ | rfc4648 | | y | -| | /base64url/ | rfc4648 | | Y | -| **Serialization formats** | -| 0x062f6a736f6e2f | /json/ | | | n/a | -| 0x062f63626f722f | /cbor/ | | | n/a | -| 0x062f62736f6e2f | /bson/ | | | n/a | -| 0x072f626a736f6e2f | /bjson/ | | | n/a | -| 0x082f75626a736f6e2f | /ubjson/ | | | n/a | -| 0x0a2f70726f746f6275662f | /protobuf/ | Protocol Buffers | | n/a | -| 0x072f6361706e702f | /capnp/ | Cap-n-Proto | | n/a | -| 0x092f666c61746275662f | /flatbuf/ | FlatBuffers | | n/a | -| 0x052f726c702f | /rlp/ | recursive length prefix | 0x60 | n/a | -| **Multiformats** | -| 0x182f6d756c7469636f6465632f | /multicodec/ | | 0x30 | n/a | -| 0x162f6d756c7469686173682f | /multihash/ | | 0x31 | n/a | -| 0x162f6d756c7469616464722f | /multiaddr/ | | 0x32 | n/a | -| **Multihashes** | -| | /sha1/ | | 0x11 | n/a | -| | /sha2-256/ | | 0x12 | n/a | -| | /sha2-512/ | | 0x13 | n/a | -| | /sha3-256/ | | 0x16 | n/a | -| | /sha3-384/ | | 0x15 | n/a | -| | /sha3-512/ | | 0x14 | n/a | -| | /shake-128/ | | 0x18 | n/a | -| | /shake-256/ | | 0x19 | n/a | -| | /blake2b/ | | 0x40 | n/a | -| | /blake2s/ | | 0x41 | n/a | -| **Multiaddrs** | -| | /ip4/ | | 0x04 | n/a | -| | /ip6/ | | 0x29 | n/a | -| | /tcp/ | | 0x06 | n/a | -| | /udp/ | | 0x11 | n/a | -| | /dccp/ | | 0x21 | n/a | -| | /sctp/ | | 0x84 | n/a | -| | /udt/ | | 0x012D | n/a | -| | /utp/ | | 0x012E | n/a | -| | /ipfs/ | | 0x2A | n/a | -| | /http/ | | 0x01E0 | n/a | -| | /https/ | | 0x01BB | n/a | -| | /ws/ | | 0x01DD | n/a | -| | /onion/ | | 0x01BC | n/a | -| **Archiving formats** | -| 0x052f7461722f | /tar/ | | | n/a | -| 0x052f7a69702f | /zip/ | | | n/a | -| **Image formats** | -| 0x052f706e672f | /png/ | | | n/a | -| | /jpg/ | | | n/a | -| **Video formats** | -| | /mp4/ | | | n/a | -| | /mkv/ | | | n/a | -| **Blockchain formats** | -| | | | | n/a | -| **VCS formats** | -| | | | | n/a | -| **IPLD formats** | -| | /dag-pb/ | MerkleDAG protobuf | | n/a | -| | /dag-cbor/ | MerkleDAG cbor | | n/a | -| | /eth-rlp/ | Ethereum Block RLP | | n/a | - -Note: Base encodings present a multibase encoding special char, so that the codec doesn't break the encoded version of the data it is describing. +| codec | description | code | +|-----------------|-------------------------|-------------| +| | +| **Miscelaneous** | +| bin | raw binary | 0x00 | +| | +| **Bases encodings** | +| base1 | unary | | +| base2 | binary (0 and 1) | 0x00 | +| base8 | octal | | +| base10 | decimal | | +| base16 | hexadecimal | | +| base32 | rfc4648 | | +| base32hex | rfc4648 | | +| base58flickr | base58 flicker | | +| base58btc | base58 bitcoin | | +| base64 | rfc4648 | | +| base64url | rfc4648 | | +| | +| **Serialization formats** | +| json | | | +| cbor | | | +| bson | | | +| bjson | | | +| ubjson | | | +| protobuf | Protocol Buffers | | +| capnp | Cap-n-Proto | | +| flatbuf | FlatBuffers | | +| rlp | recursive length prefix | 0x60 | +| | +| **Multiformats** | +| multicodec | | 0x30 | +| multihash | | 0x31 | +| multiaddr | | 0x32 | +| multibase | | 0x33 | +| | +| **Multihashes** | +| sha1 | | 0x11 | +| sha2-256 | | 0x12 | +| sha2-512 | | 0x13 | +| sha3-224 | | 0x17 | +| sha3-256 | | 0x16 | +| sha3-384 | | 0x15 | +| sha3-512 | | 0x14 | +| shake-128 | | 0x18 | +| shake-256 | | 0x19 | +| blake2b | | 0x40 | +| blake2s | | 0x41 | +| | +| **Multiaddrs** | +| ip4 | | 0x04 | +| ip6 | | 0x29 | +| tcp | | 0x06 | +| udp | | 0x11 | +| dccp | | 0x21 | +| sctp | | 0x84 | +| udt | | 0x012D | +| utp | | 0x012E | +| ipfs | | 0x2A | +| http | | 0x01E0 | +| https | | 0x01BB | +| ws | | 0x01DD | +| onion | | 0x01BC | +| | +| **Archiving formats** | +| tar | | | +| zip | | | +| | +| **Image formats** | +| png | | | +| jpg | | | +| | +| **Video formats** | +| mp4 | | | +| mkv | | | +| | +| **Blockchain formats** | +| | +| **VCS formats** | +| | +| **IPLD formats** | +| dag-pb | MerkleDAG protobuf | | +| dag-cbor | MerkleDAG cbor | | +| eth-rlp | Ethereum Block RLP | | ## Implementations -- multicodec - - [go-multicodec](https://github.com/multiformats/go-multicodec) -- multicodec-packed - - [go-multicodec-packed](https://github.com/multiformats/go-multicodec-packed) - - [js-multicodec-packed](https://github.com/multiformats/js-multicodec-packed) -- multistream - - [go-multistream](https://github.com/multiformats/go-multistream) - Implements multistream, which uses multicodec for stream negotiation - - [js-multistream](https://github.com/multiformats/js-multistream) - Implements multistream, which uses multicodec for stream negotiation - - [clj-multicodec](https://github.com/greglook/clj-multicodec) +- [go](https://github.com/multiformats/go-multicodec/) +- [JavaScript](https://github.com/multiformats/js-multicodec) +- [Add yours today!](https://github.com/multiformats/multicodec/edit/master/multicodec.md) ## FAQ +> **Q. I have questions on multicodec, not listed here.** + +That's not a question. But, have you checked the proper [multicodec FAQ](./README.md#faq)? Maybe your question is answered there. This FAQ is only specifically for multicodec-packed. + > **Q. Why?** -Today, people speak many languages, and use common ones to interface. But every "common language" has evolved over time, or even fundamentally switched. Why should we expect programs to be any different? +Because [multistream](https://github.com/multiformats/multistream) is too long for identifiers. We needed something shorter. -And the reality is they're not. Programs use a variety of encodings. Today we like JSON. Yesterday, XML was all the rage. XDR solved everything, but it's kinda retro. Protobuf is still too cool for school. capnp ("cap and proto") is -for cerealization hipsters. +> **Q. Why varints?** -The one problem is figuring out what we're speaking. Humans are pretty smart, we pick up all sorts of languages over time. And we can always resort to pointing and grunting (the ascii of humanity). +So that we have no limitation on protocols. Implementation note: you do not need to implement varints until the standard multicodec table has more than 127 functions. -Programs have a harder time. You can't keep piping json into a protobuf decoder and hope they align. So we have to help them out a bit. That's what multicodec is for. +> **Q. What kind of varints?** -> **Q. Why "codec" and not "encoder" and "decoder"?** +An Most Significant Bit unsigned varint, as defined by the [multiformats/unsigned-varint](https://github.com/multiformats/unsigned-varint). -Because they're the same thing. Which one of these is the encoder and which the decoder? +> **Q. Don't we have to agree on a table of protocols?** - 5555 ----[ THING ]---> 8888 - 5555 <---[ THING ]---- 8888 - -> **Q. Full paths are too big for my use case, is there something smaller?** - -Yes, check out [multicodec-packed](./multicodec-packed.md). It uses a varint and a table to achieve the same thing. +Yes, but we already have to agree on what protocols themselves are, so this is not so hard. The table even leaves some room for custom protocol paths, or you can use your own tables. The standard table is only for common things. ## Maintainers @@ -222,5 +178,3 @@ Check out our [contributing document](https://github.com/multiformats/multiforma ## License [MIT](LICENSE) - -[p](https://github.com/multiformats/multicodec/blob/master/multicodec-packed.md) diff --git a/multicodec-packed.md b/multicodec-packed.md deleted file mode 100644 index f338872..0000000 --- a/multicodec-packed.md +++ /dev/null @@ -1,93 +0,0 @@ -# multicodec-packed - -[![](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://ipn.io) -[![](https://img.shields.io/badge/project-multiformats-blue.svg?style=flat-square)](http://github.com/multiformats/multiformats) -[![](https://img.shields.io/badge/freenode-%23ipfs-blue.svg?style=flat-square)](http://webchat.freenode.net/?channels=%23ipfs) - -> compact self-describing codecs. Save space by using predefined multicodec tables. - -## Table of Contents - -- [Motivation](#motivation) -- [How does it work? - Protocol Description](#how-does-it-work---protocol-description) -- [Multicodec-Packed Protocol Tables](#multicodec-packed-protocol-tables) - - [Standard mcp protocol table](#standard-mcp-protocol-table) -- [Implementations](#implementations) -- [FAQ](#faq) -- [Maintainers](#maintainers) -- [Contribute](#contribute) -- [License](#license) - -## Motivation - -[Multicodecs](./README.md) are self-describing protocol/encoding streams. Multicodec-packed is a different representation of multicodec, which uses an agreed-upon "protocol table". It is designed for use in short strings, such as keys or identifiers (i.e [CID](https://github.com/ipld/cid)). - -## Protocol Description - How does the protocol work? - -`multicodec-packed` is a _self-describing multiformat_, it wraps other formats with a tiny bit of self-description. A multicodec-packed identifier is both a varint and the code identifying the following data, this means that the most significant bit of every multicodec-packed code is reserved to signal the continuation. - -This way, a chunk of data identified by multicodec will look like this: - -```sh - -# To reduce the cognitive load, we sometimes might write the same line as: - -``` - -Another useful scenario is when using the multicodec-packed as part of the keys to access data, example: - -``` -# suppose we have a value and a key to retrieve it -"" -> - -# we can use multicodec-packed with the key to know what codec the value is in -"" -> -``` - -It is worth noting that multicodec-packed works very well in conjunction with [multihash](https://github.com/multiformats/multihash) and [multiaddr](https://github.com/multiformats/multiaddr), as you can prefix those values with a multicodec-packed to tell what they are. - -## Multicodec-Packed Protocol Tables - -Multicodec-packed uses "protocol tables" to agree upon the mapping from one multicodec-packed code (a single varint). These tables map an `` to a full [multicodec protocol path](./README.md#the-protocol-path). These tables can be application specific, though -- like [with](https://github.com/multiformats/multihash) [other](https://github.com/multiformats/multibase) [multiformats](https://github.com/multiformats/multiaddr) -- we will keep a globally agreed upon table with common protocols and formats. - -## Implementations - -- [go](https://github.com/multiformats/go-multicodec-packed/) -- [JavaScript](https://github.com/multiformats/js-multicodec-packed) -- [Add yours today!](https://github.com/multiformats/multicodec/edit/master/multicodec-packed.md) - -## FAQ - -> **Q. I have questions on multicodec, not listed here.** - -That's not a question. But, have you checked the proper [multicodec FAQ](./README.md#faq)? Maybe your question is answered there. This FAQ is only specifically for multicodec-packed. - -> **Q. Why?** - -Because [multicodec](./README.md) is too long for identifiers. We needed something shorter. - -> **Q. Why varints?** - -So that we have no limitation on protocols. Implementation note: you do not need to implement varints until the standard multicodec table has more than 127 functions. - -> **Q. What kind of varints?** - -An Most Significant Bit unsigned varint, as defined by the [multiformats/unsigned-varint](https://github.com/multiformats/unsigned-varint). - -> **Q. Don't we have to agree on a table of protocols?** - -Yes, but we already have to agree on what protocols themselves are, so this is not so hard. The table even leaves some room for custom protocol paths, or you can use your own tables. The standard table is only for common things. - -## Maintainers - -Captain: [@jbenet](https://github.com/jbenet). - -## Contribute - -Contributions welcome. Please check out [the issues](https://github.com/multiformats/multicodec/issues). - -Check out our [contributing document](https://github.com/multiformats/multiformats/blob/master/contributing.md) for more information on how we work, and about contributing in general. Please be aware that all interactions related to multiformats are subject to the IPFS [Code of Conduct](https://github.com/ipfs/community/blob/master/code-of-conduct.md). - -## License - -[MIT](LICENSE) From a3fc375fc7315b09076b63279f174a4615fb5a13 Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 26 Sep 2016 17:28:22 +0800 Subject: [PATCH 06/17] move to csv --- README.md | 169 +++++++++++++++++++++++++++--------------------------- 1 file changed, 85 insertions(+), 84 deletions(-) diff --git a/README.md b/README.md index 692c256..38fc511 100644 --- a/README.md +++ b/README.md @@ -52,90 +52,91 @@ Multicodecuses "protocol tables" to agree upon the mapping from one multicodec c ## Multicodec table -| codec | description | code | -|-----------------|-------------------------|-------------| -| | -| **Miscelaneous** | -| bin | raw binary | 0x00 | -| | -| **Bases encodings** | -| base1 | unary | | -| base2 | binary (0 and 1) | 0x00 | -| base8 | octal | | -| base10 | decimal | | -| base16 | hexadecimal | | -| base32 | rfc4648 | | -| base32hex | rfc4648 | | -| base58flickr | base58 flicker | | -| base58btc | base58 bitcoin | | -| base64 | rfc4648 | | -| base64url | rfc4648 | | -| | -| **Serialization formats** | -| json | | | -| cbor | | | -| bson | | | -| bjson | | | -| ubjson | | | -| protobuf | Protocol Buffers | | -| capnp | Cap-n-Proto | | -| flatbuf | FlatBuffers | | -| rlp | recursive length prefix | 0x60 | -| | -| **Multiformats** | -| multicodec | | 0x30 | -| multihash | | 0x31 | -| multiaddr | | 0x32 | -| multibase | | 0x33 | -| | -| **Multihashes** | -| sha1 | | 0x11 | -| sha2-256 | | 0x12 | -| sha2-512 | | 0x13 | -| sha3-224 | | 0x17 | -| sha3-256 | | 0x16 | -| sha3-384 | | 0x15 | -| sha3-512 | | 0x14 | -| shake-128 | | 0x18 | -| shake-256 | | 0x19 | -| blake2b | | 0x40 | -| blake2s | | 0x41 | -| | -| **Multiaddrs** | -| ip4 | | 0x04 | -| ip6 | | 0x29 | -| tcp | | 0x06 | -| udp | | 0x11 | -| dccp | | 0x21 | -| sctp | | 0x84 | -| udt | | 0x012D | -| utp | | 0x012E | -| ipfs | | 0x2A | -| http | | 0x01E0 | -| https | | 0x01BB | -| ws | | 0x01DD | -| onion | | 0x01BC | -| | -| **Archiving formats** | -| tar | | | -| zip | | | -| | -| **Image formats** | -| png | | | -| jpg | | | -| | -| **Video formats** | -| mp4 | | | -| mkv | | | -| | -| **Blockchain formats** | -| | -| **VCS formats** | -| | -| **IPLD formats** | -| dag-pb | MerkleDAG protobuf | | -| dag-cbor | MerkleDAG cbor | | -| eth-rlp | Ethereum Block RLP | | +```csv +codec, description, code + +miscelaneous +bin, raw binary, 0x00 + +bases encodings +base1, unary, 0x01 +base2, binary (0 and 1), 0x00 +base8, octal, 0x07 +base10, decimal, 0x09 +base16, hexadecimal, 0x +base32, rfc4648, 0x +base32hex, rfc4648, 0x +base58flickr, base58 flicker, 0x +base58btc, base58 bitcoin, 0x +base64, rfc4648, 0x +base64url, rfc4648, 0x + +serialization formats +json, , 0x +cbor, , 0x +bson, , 0x +bjson, , 0x +ubjson, , 0x +protobuf, Protocol Buffers, 0x +capnp, Cap-n-Proto, 0x +flatbuf, FlatBuffers, 0x +rlp, recursive length prefix, 0x60 + +multiformats +multicodec, , 0x30 +multihash, , 0x31 +multiaddr, , 0x32 +multibase, , 0x33 + +multihashes +sha1, , 0x11 +sha2-256, , 0x12 +sha2-512, , 0x13 +sha3-224, , 0x17 +sha3-256, , 0x16 +sha3-384, , 0x15 +sha3-512, , 0x14 +shake-128, , 0x18 +shake-256, , 0x19 +blake2b, , 0x40 +blake2s, , 0x41 + +multiaddrs +ip4, , 0x04 +ip6, , 0x29 +tcp, , 0x06 +udp, , 0x11 +dccp, , 0x21 +sctp, , 0x84 +udt, , 0x012D +utp, , 0x012E +ipfs, , 0x2A +http, , 0x01E0 +https, , 0x01BB +ws, , 0x01DD +onion, , 0x01BC + +archiving formats +tar, , 0x +zip, , 0x + +image formats +png, , 0x +jpg, , 0x + +video formats +mp4, , 0x +mkv, , 0x + +blockchain formats + +VCS formats + +IPLD formats +dag-pb, MerkleDAG protobuf, 0x +dag-cbor, MerkleDAG cbor, 0x +eth-rlp, Ethereum Block RLP, 0x +``` ## Implementations From 90aa4fb43d76f2259065e4e6924bae5d0012451f Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 28 Sep 2016 00:32:09 +0800 Subject: [PATCH 07/17] allocated codecs for IPLD formats --- README.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 38fc511..39ed26c 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,7 @@ shake-128, , 0x18 shake-256, , 0x19 blake2b, , 0x40 blake2s, , 0x41 +reserved for apps, appl specific range, 0x4000-0x40f multiaddrs ip4, , 0x04 @@ -133,9 +134,14 @@ blockchain formats VCS formats IPLD formats -dag-pb, MerkleDAG protobuf, 0x -dag-cbor, MerkleDAG cbor, 0x -eth-rlp, Ethereum Block RLP, 0x +dag-pb, MerkleDAG protobuf, 0x70 +dag-cbor, MerkleDAG cbor, 0x71 +eth-block, Ethereum Block (RLP), 0x90 +eth-tx, Ethereum Tx (RLP), 0x91 +bitcoin-block, Bitcoin Block, 0xb0 +bitcoin-tx, Bitcoin Tx, 0xb1 +stellar-block, Stellar Block, 0xd0 +stellar-tx, Stellar Tx, 0xd1 ``` ## Implementations From 885959da0cfffe5c8961d2238c357a955b66a527 Mon Sep 17 00:00:00 2001 From: kumavis Date: Thu, 20 Oct 2016 15:25:29 -0700 Subject: [PATCH 08/17] Add keccak-* keccak-256: 0x1A keccak-448: 0x1B keccak-512: 0x1C keccak-768: 0x1D keccak-1024: 0x1E --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 39ed26c..71822e1 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,11 @@ sha3-384, , 0x15 sha3-512, , 0x14 shake-128, , 0x18 shake-256, , 0x19 +keccak-256, , 0x1A +keccak-448, , 0x1B +keccak-512, , 0x1C +keccak-768, , 0x1D +keccak-1024, , 0x1E blake2b, , 0x40 blake2s, , 0x41 reserved for apps, appl specific range, 0x4000-0x40f From 38ea31b4839641f05d0f87f628ee96152bc5228c Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 24 Oct 2016 13:09:30 +0200 Subject: [PATCH 09/17] add missing 0 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 71822e1..24014cb 100644 --- a/README.md +++ b/README.md @@ -105,7 +105,7 @@ keccak-768, , 0x1D keccak-1024, , 0x1E blake2b, , 0x40 blake2s, , 0x41 -reserved for apps, appl specific range, 0x4000-0x40f +reserved for apps, appl specific range, 0x4000-0x40f0 multiaddrs ip4, , 0x04 From 76ed15d1642eb0bccd803fcf66081e661663cd09 Mon Sep 17 00:00:00 2001 From: kumavis Date: Wed, 26 Oct 2016 11:06:00 -0700 Subject: [PATCH 10/17] Fix keccak-* naming --- README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 24014cb..6da1544 100644 --- a/README.md +++ b/README.md @@ -98,11 +98,10 @@ sha3-384, , 0x15 sha3-512, , 0x14 shake-128, , 0x18 shake-256, , 0x19 -keccak-256, , 0x1A -keccak-448, , 0x1B -keccak-512, , 0x1C -keccak-768, , 0x1D -keccak-1024, , 0x1E +keccak-224, , 0x1A +keccak-256, , 0x1B +keccak-384, , 0x1C +keccak-512, , 0x1D blake2b, , 0x40 blake2s, , 0x41 reserved for apps, appl specific range, 0x4000-0x40f0 From 36bd99e827a67405bbbfc0387b5b9a7567d77c50 Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 7 Nov 2016 20:40:58 +0000 Subject: [PATCH 11/17] docs: note about keccak --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6da1544..50ebb29 100644 --- a/README.md +++ b/README.md @@ -46,9 +46,9 @@ Another useful scenario is when using the multicodec-packed as part of the keys It is worth noting that multicodec-packed works very well in conjunction with [multihash](https://github.com/multiformats/multihash) and [multiaddr](https://github.com/multiformats/multiaddr), as you can prefix those values with a multicodec-packed to tell what they are. -## Multicodec-Packed Protocol Tables +## MulticodecProtocol Tables -Multicodecuses "protocol tables" to agree upon the mapping from one multicodec code (a single varint). These tables can be application specific, though -- like [with](https://github.com/multiformats/multihash) [other](https://github.com/multiformats/multibase) [multiformats](https://github.com/multiformats/multiaddr) -- we will keep a globally agreed upon table with common protocols and formats. +Multicodec uses "protocol tables" to agree upon the mapping from one multicodec code (a single varint). These tables can be application specific, though -- like [with](https://github.com/multiformats/multihash) [other](https://github.com/multiformats/multibase) [multiformats](https://github.com/multiformats/multiaddr) -- we will keep a globally agreed upon table with common protocols and formats. ## Multicodec table @@ -102,6 +102,7 @@ keccak-224, , 0x1A keccak-256, , 0x1B keccak-384, , 0x1C keccak-512, , 0x1D +Note: keccak has variable output length, instead the number specifies the core length blake2b, , 0x40 blake2s, , 0x41 reserved for apps, appl specific range, 0x4000-0x40f0 @@ -148,6 +149,7 @@ stellar-block, Stellar Block, 0xd0 stellar-tx, Stellar Tx, 0xd1 ``` + ## Implementations - [go](https://github.com/multiformats/go-multicodec/) From 66dcfa4fd40c20d50c7ff6b9d956899c5b11360c Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 7 Nov 2016 20:41:26 +0000 Subject: [PATCH 12/17] move binary to 0x55 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 50ebb29..f56de9a 100644 --- a/README.md +++ b/README.md @@ -56,7 +56,7 @@ Multicodec uses "protocol tables" to agree upon the mapping from one multicodec codec, description, code miscelaneous -bin, raw binary, 0x00 +bin, raw binary, 0x55 bases encodings base1, unary, 0x01 From fa3197fbf890ac28b4b5f3fdde6cbcb76d7e66f1 Mon Sep 17 00:00:00 2001 From: David Dias Date: Mon, 7 Nov 2016 21:25:15 +0000 Subject: [PATCH 13/17] docs: reference multistream --- README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index f56de9a..6842ea9 100644 --- a/README.md +++ b/README.md @@ -102,7 +102,7 @@ keccak-224, , 0x1A keccak-256, , 0x1B keccak-384, , 0x1C keccak-512, , 0x1D -Note: keccak has variable output length, instead the number specifies the core length +Note: keccak has variable output length, instead the number specifies the core length,, blake2b, , 0x40 blake2s, , 0x41 reserved for apps, appl specific range, 0x4000-0x40f0 @@ -149,13 +149,18 @@ stellar-block, Stellar Block, 0xd0 stellar-tx, Stellar Tx, 0xd1 ``` - ## Implementations - [go](https://github.com/multiformats/go-multicodec/) - [JavaScript](https://github.com/multiformats/js-multicodec) - [Add yours today!](https://github.com/multiformats/multicodec/edit/master/multicodec.md) +## Multicodec Path, also known as [`multistream`](https://github.com/multiformats/multistream) + +Multicodec defines a table for the most common data serialization formats that can be expanded overtime or per application bases, however, in order for two programs to talk with each other, they need to know before hand which table or table extension is being used. + +In order to enable self descriptive data formats or streams that can be dynamically described, without the formal set of adding a binary packed code to a table, we have [`multistream`](https://github.com/multiformats/multistream), so that applications can adopt multiple data formats for their streams and with that create different protocols. + ## FAQ > **Q. I have questions on multicodec, not listed here.** From ab1b1964680618c74a3b6ee7869612c937458bde Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 9 Nov 2016 13:07:57 +0000 Subject: [PATCH 14/17] update codec table --- README.md | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 6842ea9..d1a2fb2 100644 --- a/README.md +++ b/README.md @@ -60,7 +60,7 @@ bin, raw binary, 0x55 bases encodings base1, unary, 0x01 -base2, binary (0 and 1), 0x00 +base2, binary (0 and 1), 0x55 base8, octal, 0x07 base10, decimal, 0x09 base16, hexadecimal, 0x @@ -72,11 +72,9 @@ base64, rfc4648, 0x base64url, rfc4648, 0x serialization formats -json, , 0x -cbor, , 0x -bson, , 0x -bjson, , 0x -ubjson, , 0x +cbor, CBOR, 0x +bson, Binary JSON, 0x +ubjson, Universal Binary JSON, 0x protobuf, Protocol Buffers, 0x capnp, Cap-n-Proto, 0x flatbuf, FlatBuffers, 0x @@ -134,10 +132,6 @@ video formats mp4, , 0x mkv, , 0x -blockchain formats - -VCS formats - IPLD formats dag-pb, MerkleDAG protobuf, 0x70 dag-cbor, MerkleDAG cbor, 0x71 From ce943345e96c61c402d927035dc7885fe06da593 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 9 Nov 2016 13:12:13 +0000 Subject: [PATCH 15/17] resolve udp sha1 clash --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d1a2fb2..53fc247 100644 --- a/README.md +++ b/README.md @@ -109,7 +109,7 @@ multiaddrs ip4, , 0x04 ip6, , 0x29 tcp, , 0x06 -udp, , 0x11 +udp, , 0x0011 dccp, , 0x21 sctp, , 0x84 udt, , 0x012D From 6d7f49dbc4059f010a54ea534c739e52f8797900 Mon Sep 17 00:00:00 2001 From: David Dias Date: Wed, 9 Nov 2016 15:04:03 +0000 Subject: [PATCH 16/17] move udp from 0x0011 to 0x0111 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 53fc247..23147a0 100644 --- a/README.md +++ b/README.md @@ -109,7 +109,7 @@ multiaddrs ip4, , 0x04 ip6, , 0x29 tcp, , 0x06 -udp, , 0x0011 +udp, , 0x0111 dccp, , 0x21 sctp, , 0x84 udt, , 0x012D From 0c9d2dfbea966f8f758eaf0a5e44692a0be01009 Mon Sep 17 00:00:00 2001 From: David Dias Date: Sun, 13 Nov 2016 05:23:37 +0000 Subject: [PATCH 17/17] break multicodec table into its own file, add a note of how to add new codes --- README.md | 93 +++++-------------------------------------------------- table.csv | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+), 86 deletions(-) create mode 100644 table.csv diff --git a/README.md b/README.md index 23147a0..b340884 100644 --- a/README.md +++ b/README.md @@ -52,96 +52,17 @@ Multicodec uses "protocol tables" to agree upon the mapping from one multicodec ## Multicodec table -```csv -codec, description, code +The full table can be found at [table.csv](/table.csv) inside this repo. -miscelaneous -bin, raw binary, 0x55 +### Adding new multicodecs to the table -bases encodings -base1, unary, 0x01 -base2, binary (0 and 1), 0x55 -base8, octal, 0x07 -base10, decimal, 0x09 -base16, hexadecimal, 0x -base32, rfc4648, 0x -base32hex, rfc4648, 0x -base58flickr, base58 flicker, 0x -base58btc, base58 bitcoin, 0x -base64, rfc4648, 0x -base64url, rfc4648, 0x +The process to add a new multicodec to the table is the following: -serialization formats -cbor, CBOR, 0x -bson, Binary JSON, 0x -ubjson, Universal Binary JSON, 0x -protobuf, Protocol Buffers, 0x -capnp, Cap-n-Proto, 0x -flatbuf, FlatBuffers, 0x -rlp, recursive length prefix, 0x60 +- 1. Fork this repo +- 2. Update the table with the value you want to add +- 3. Submit a Pull Request -multiformats -multicodec, , 0x30 -multihash, , 0x31 -multiaddr, , 0x32 -multibase, , 0x33 - -multihashes -sha1, , 0x11 -sha2-256, , 0x12 -sha2-512, , 0x13 -sha3-224, , 0x17 -sha3-256, , 0x16 -sha3-384, , 0x15 -sha3-512, , 0x14 -shake-128, , 0x18 -shake-256, , 0x19 -keccak-224, , 0x1A -keccak-256, , 0x1B -keccak-384, , 0x1C -keccak-512, , 0x1D -Note: keccak has variable output length, instead the number specifies the core length,, -blake2b, , 0x40 -blake2s, , 0x41 -reserved for apps, appl specific range, 0x4000-0x40f0 - -multiaddrs -ip4, , 0x04 -ip6, , 0x29 -tcp, , 0x06 -udp, , 0x0111 -dccp, , 0x21 -sctp, , 0x84 -udt, , 0x012D -utp, , 0x012E -ipfs, , 0x2A -http, , 0x01E0 -https, , 0x01BB -ws, , 0x01DD -onion, , 0x01BC - -archiving formats -tar, , 0x -zip, , 0x - -image formats -png, , 0x -jpg, , 0x - -video formats -mp4, , 0x -mkv, , 0x - -IPLD formats -dag-pb, MerkleDAG protobuf, 0x70 -dag-cbor, MerkleDAG cbor, 0x71 -eth-block, Ethereum Block (RLP), 0x90 -eth-tx, Ethereum Tx (RLP), 0x91 -bitcoin-block, Bitcoin Block, 0xb0 -bitcoin-tx, Bitcoin Tx, 0xb1 -stellar-block, Stellar Block, 0xd0 -stellar-tx, Stellar Tx, 0xd1 -``` +This ["first come, first assign"](https://github.com/multiformats/multicodec/pull/16#issuecomment-260146609) policy is a way to assign codes as they are most needed, without increasing the size of the table (and therefore the size of the multicodecs) too rapidly. ## Implementations diff --git a/table.csv b/table.csv new file mode 100644 index 0000000..2a8ee27 --- /dev/null +++ b/table.csv @@ -0,0 +1,88 @@ +codec, description, code + +miscelaneous,, +bin, raw binary, 0x55 + +bases encodings,, +base1, unary, 0x01 +base2, binary (0 and 1), 0x55 +base8, octal, 0x07 +base10, decimal, 0x09 +base16, hexadecimal, 0x +base32, rfc4648, 0x +base32hex, rfc4648, 0x +base58flickr, base58 flicker, 0x +base58btc, base58 bitcoin, 0x +base64, rfc4648, 0x +base64url, rfc4648, 0x + +serialization formats,, +cbor, CBOR, 0x +bson, Binary JSON, 0x +ubjson, Universal Binary JSON, 0x +protobuf, Protocol Buffers, 0x +capnp, Cap-n-Proto, 0x +flatbuf, FlatBuffers, 0x +rlp, recursive length prefix, 0x60 + +multiformats,, +multicodec, , 0x30 +multihash, , 0x31 +multiaddr, , 0x32 +multibase, , 0x33 + +multihashes,, +sha1, , 0x11 +sha2-256, , 0x12 +sha2-512, , 0x13 +sha3-224, , 0x17 +sha3-256, , 0x16 +sha3-384, , 0x15 +sha3-512, , 0x14 +shake-128, , 0x18 +shake-256, , 0x19 +keccak-224, , 0x1A +keccak-256, , 0x1B +keccak-384, , 0x1C +keccak-512, , 0x1D +,, Note: keccak has variable output length. The number specifies the core length +blake2b, , 0x40 +blake2s, , 0x41 +reserved for apps, appl specific range, 0x4000-0x40f0 + +multiaddrs,, +ip4, , 0x04 +ip6, , 0x29 +tcp, , 0x06 +udp, , 0x0111 +dccp, , 0x21 +sctp, , 0x84 +udt, , 0x012D +utp, , 0x012E +ipfs, , 0x2A +http, , 0x01E0 +https, , 0x01BB +ws, , 0x01DD +onion, , 0x01BC + +archiving formats,, +tar, , 0x +zip, , 0x + +image formats,, +png, , 0x +jpg, , 0x + +video formats,, +mp4, , 0x +mkv, , 0x + +IPLD formats,, +dag-pb, MerkleDAG protobuf, 0x70 +dag-cbor, MerkleDAG cbor, 0x71 +eth-block, Ethereum Block (RLP), 0x90 +eth-tx, Ethereum Tx (RLP), 0x91 +bitcoin-block, Bitcoin Block, 0xb0 +bitcoin-tx, Bitcoin Tx, 0xb1 +stellar-block, Stellar Block, 0xd0 +stellar-tx, Stellar Tx, 0xd1