bittorrent-codex-docs/10 Notes/Advertising BitTorrent content on Codex.md

tags	related-to	related
bittorrent	How BitTorrent-Codex integration may look like? Learn BitTorrent	Uploading and downloading content in Codex

#bittorrent

related-to	How BitTorrent-Codex integration may look like?, Learn BitTorrent
related	Uploading and downloading content in Codex

This content builds upon BitTorrent metadata files

Let's start with BitTorrent protocol version 1 and let's start with relatively simple case: a single 40 kiB single file:

! We have a single, incomplete piece of size 64 kiB, which will be transferred as 3 blocks (3rd block will be shorter, namely 8 kiB). Peers in BitTorrent v1, operate at the piece level when requesting the content. The request message contains an index, begin, and length. Except when it is truncated at the end of the file, the length is generally power of 2 and all current implementations use 2^{14} = 16 kiB and close connections which request an amount greater than that.

The corresponding torrent file will be the following:

{
  "announce": "http://example.com/announce",
  "info": {
    "length": 40960,
    "name": "data40k.bin",
    "piece length": 65536,
    "pieces": [
      "1cc46da027e7ff6f1970a2e58880dbc6a08992a0"
    ]
  }
}

or in the b-encoded form:

d8:announce27:http://example.com/announce4:infod6:lengthi40960e4:name11:data40k.bin12:piece lengthi65536e6:pieces20:bbbbbbbbbbbbbbbbbbbb

where bbbbbbbbbbbbbbbbbbbb stands for 20 successive bytes being sha1 hash of the only piece. When computing the hash, no padding was used at the end of the file:

openssl sha1 experiment-3/data40k.bin 
SHA1(experiment-3/data40k.bin)= 1cc46da027e7ff6f1970a2e58880dbc6a08992a0

The info hash (sha1) of the bencoded info dictionary is: 1902d602db8c350f4f6d809ed01eff32f030da95.

Now, let's assume our input is a magnet link corresponding to the above torrent file:

magnet:?xt=urn:btih:1902d602db8c350f4f6d809ed01eff32f030da95&dn=data40k.bin

The magnet link itself does not tell us details of the corresponding torrent file: what we have is only the hash of the b-encoded info directory. How does the peer interested in downloading the content designated by the given magnet link finds the remaining details of the info dictionary? This is defined in BEP9 - Extension for Peers to Send Metadata Files.

Note

BEP9 is also where the magnet links were introduced.

BEP9 - Extension for Peers to Send Metadata Files uses BEP10 Extension Protocol as a “carrier”. The metadata is handled in blocks of 16KiB (16384 Bytes). The metadata blocks are indexed starting at 0. All blocks are 16KiB except the last block which may be smaller. The details of BEP9 - Extension for Peers to Send Metadata Files and BEP10 Extension Protocol are less important here, only when necessary for a better understanding of what has to be considered as an addition to the Codex protocol that we will mention some details where deemed necessary.

In version 1 of the BitTorrent protocol, we can only check the integrity of the data being downloaded at the piece level. In our example, there is only one piece, and the downloading peer will need three request messages in order to fetch the necessary data:

1. { index=0, being=0, length=16384 }
2. { index=0, being=16384, length=16384 }
3. { index=0, being=32768, length=8192 }

Thus, two blocks of 16 kiB and one shorter of 8 kiB. Only after we fetch the whole piece, we will be able to compute and verify its hash.

When having multiple files, the situation does not change much at least form the perspective of Codex integration. Let’s looks at the example, which additionally will help us to clarify some interesting facts about piece alignment extensions and hybrid torrents.

Here we have an example with three files:

• 258 kiB (named data258k.bin)
• 40 kiB (named data40k.bin)
• 72 kiB (named data72k.bin)

Following the original BEP3 - The BitTorrent Protocol Specification, the files would be ordered in sequence one after and projected on the pieces spaces without any alignment. This would make, however, interoperability with the version 2 of the protocol harder. Normally, the same content will be represented by different info hashes for different versions of the protocol and thus the peers using info hash for version 1 of the protocol will form a separate swarm from those peers using info hash for version 2. In some circumstances, it may be beneficial for the peer to join both swarms in order to complete the download faster as some content may be available in the swarm for version 1 info hash while not being available in the swarm for version 2. To keep higher level of the protocols agnostic to which version of the protocol is used to acquire relevant pieces, it is desired that the content in both swarms has the same file names, order, and piece alignment so that the same request message can be used for both swarm resulting in the same bytes being downloaded. This is where BEP47 - Padding files and extended file attributes comes handy. BEP47 - Padding files and extended file attributes allows the files in version 1 of the protocol to also be aligned on the piece boundary by introducing padding. This is what we see in the picture below.

!

File data258k.bin is “sticking out” of the piece boundary by exactly 2 kiB. In order to make sure that the next file is aligned to the following piece, a padding file of 62 kiB is added. Similarly, file 40 kiB file data40k.bin will be padded with 24576 Bytes (24 kiB) and the last file will be padded with 56 kiB.

Note

The last file is also padded to the piece boundary (although I do not see any good reason for that).

Below is the corresponding torrent file:

{
  "announce": "http://example.com/announce",
  "info": {
    "files": [
      {
        "length": 264192,
        "path": [
          "data258k.bin"
        ]
      },
      {
        "attr": "p",
        "length": 63488,
        "path": [
          ".pad",
          "63488"
        ]
      },
      {
        "length": 40960,
        "path": [
          "data40k.bin"
        ]
      },
      {
        "attr": "p",
        "length": 24576,
        "path": [
          ".pad",
          "24576"
        ]
      },
      {
        "length": 73728,
        "path": [
          "data72k.bin"
        ]
      },
      {
        "attr": "p",
        "length": 57344,
        "path": [
          ".pad",
          "57344"
        ]
      }
    ],
    "name": "experiment-6",
    "piece length": 65536,
    "pieces": [
      "6e2f08ac8c18ac0e1a639474781882019bac40fd",
      "52422fcc223e90b84a3b60acbdee6104e2833234",
      "3c1a3b20f63ada631660c9416ec545ccc0995797",
      "8e59fe5e59b32fc58f6d0d44355f01cbc0e5a37a",
      "9772758f13faf635e02fecb47df4c25aed237981",
      "145355c75352ccc9a8500fcc6e8d21a65e4380b2",
      "19cc13b4954cc65b8de2578a6004d383947fa0dc",
      "92e3dbd456711254df6feddcb6f0b8b1aa7f53bf"
    ]
  }
}

Padding files have their own names, and are included in the files entry of the info dictionary. Clients aware of BEP47 - Padding files and extended file attributes extension don't need to write the padding files to disk and should also avoid requesting byte-ranges covering their contents, e.g. via request messages. But for backwards-compatibility they must service such requests. For the calculation of piece hashes the content of padding file is all zeros.

Having this content alignment in place, a peer having a hybrid magnet link, i.e. a link with both version 1 and version 2 hashes, may try to join both version 1 and version 2 swarms for a faster content download.

Hybrid torrents seem a bit under-documented. I am trying to gather some thoughts about them in Hybrid Torrents.

To summarize, for trackerless version 1 torrents, having only info hash as the input, in order to be able to successfully download and verify the integrity of the content, the peers need to support BEP9 - Extension for Peers to Send Metadata Files.

How the peer knows it gets the right metadata

The extension only sends the content of the info dictionary. For a good reason. Anything outside of the info dictionary cannot be validated, as the corresponding info hash covers only the info entry of the torrent file. So, for example, for our single file torrent:

{
  "announce": "http://example.com/announce",
  "info": {
    "length": 40960,
    "name": "data40k.bin",
    "piece length": 65536,
    "pieces": [
      "1cc46da027e7ff6f1970a2e58880dbc6a08992a0"
    ]
  }
}

The info dictionary is:

{
  "length": 40960,
  "name": "data40k.bin",
  "piece length": 65536,
  "pieces": [
    "1cc46da027e7ff6f1970a2e58880dbc6a08992a0"
  ]
}

or bencoded (for convenience I am using Python notation, where b"some-data" is a byte object or a sequence of bytes):

b"d6:lengthi40960e4:name11:data40k.bin12:piece lengthi65536e6:pieces20:\x1c\xc4m\xa0'\xe7\xffo\x19p\xa2\xe5\x88\x80\xdb\xc6\xa0\x89\x92\xa0e"

This is a short info dictionary, thus the client would only request a single piece (incomplete 16 kiB block - 90 bytes in this case). A peer, after indicating it support the BEP10 Extension Protocol extension protocol (by setting bit 4 of reserved_byte[5] to 1) during the general handshake, it will send the following BEP9 extension handshake message:

{b'm': {b'ut_metadata', 3}, b'metadata_size': 90}

It will be bencoded and the complete extension handshake message (including length prefix (uint32_t, big endian), message id (uint8_t), extended message id (uint8_t) and payload) will be represented by the following byte object:

from bep_0052_torrent_creator import encode
handshakePayload = encode({b'm': {b'ut_metadata': 3}, b'metadata_size': 90})
# b'd1:md11:ut_metadatai3ee13:metadata_sizei90ee'
assert len(handshakePayload) == 44
extensionHandshake = b'\x00\x00\x00\x32\x16\x00d1:md11:ut_metadatai3ee13:metadata_sizei90ee'
assert len(extensionHandshake) == 50 # 0x32 in extensionHandshake[3]

Then we will have metadata request:

requestPayload = {b'msg_type': 0, b'piece': 0}
requestBenc = encode(requestPayload)
assert requestBenc == b'd8:msg_typei0e5:piecei0ee'
assert len(requestBenc) == 25
metadataRequest = b'\x00\x00\x00\x1F\x16\x03d8:msg_typei0e5:piecei0ee'
assert len(metadataRequest) == 31 # 0x1F in extensionHandshake[3]

Finally the response will be:

responsePayload = {b'msg_type': 1, b'piece': 0, b'total_size': 90}
responseBenc = encode(responsePayload)
assert responseBenc == b'd8:msg_typei1e5:piecei0e10:total_sizei90ee'
assert len(responseBenc) == 42
metadataResponse = metadataResponse = b"\x00\x00\x00\x8A\x16\x03d8:msg_typei1e5:piecei0e10:total_sizei90eed6:lengthi40960e4:name11:data40k.bin12:piece lengthi65536e6:pieces20:\x1c\xc4m\xa0'\xe7\xffo\x19p\xa2\xe5\x88\x80\xdb\xc6\xa0\x89\x92\xa0e"
assert len(metadataResponse) == 138 # 0x8A in extensionHandshake[3]

After receiving metadata, the peer can compute hash over the bencoded response payload and compare it with the info hash from the magnet link:

assert sha1(bencoded).hexdigest() == '1902d602db8c350f4f6d809ed01eff32f030da95'

Having metadata, we can now verify each retrieved piece.

BitTorrent Version 2

In BitTorrent version 2, our input will be version 2 magnet link contained tagged tagged-info-hash, which is the multihash formatted, hex encoded full info hash for torrents in the new metadata format. For example, for our three files contents from the previous example the sha256 of the bencoded info dictionary will be:

970603312f21c543826c3bad8e289de8d68678298701b8579ce448895ce6dcd6

and as multihash it will get 1220 prefix:

1220970603312f21c543826c3bad8e289de8d68678298701b8579ce448895ce6dcd6

The v2 torrent file can be the following:

{
  "announce": "http://example.com/announce",
  "info": {
    "file tree": {
      "data258k.bin": {
        "": {
          "length": 264192,
          "pieces root": "d62f5c8510048ba73a1950a6d27750c6262f88be2016f8f9d4b2b9ffe51477e1"
        }
      },
      "data40k.bin": {
        "": {
          "length": 40960,
          "pieces root": "703ef11e93d8ef1f052abace41e3e40fa99842697b405e4c744abad11017a11a"
        }
      },
      "data72k.bin": {
        "": {
          "length": 73728,
          "pieces root": "857663dce7d614983b289c0939513130c0bfd451447268655413b5a6b72a593c"
        }
      }
    },
    "meta version": 2,
    "name": "experiment-6",
    "piece length": 65536
  },
  "piece layers": {
    "857663dce7d614983b289c0939513130c0bfd451447268655413b5a6b72a593c": "bfff3a7f25089296b73b2cb80c48cf799c858e043e52f4a6326e938e900bb0e9b4b42d885b50f3d447ffd4a17905d5d03a0495270a71177d41155e86a03d9bdb",
    "d62f5c8510048ba73a1950a6d27750c6262f88be2016f8f9d4b2b9ffe51477e1": "52a914929e9bde10dfc1f80141d15c82c8c092693cc3eab03139e24d69e9e305ffc938c3889eb4a7168894673d525f983b3749b1ae19bdd645fdffe5dbcf043b8186be160cf2774d92786db55a0d7e3a960a9f02f52bdd801573bc81b7c2152b5f22fe59816961da2f451afa606ece30ce9ec7036423bfd14adbb6fb2a17559a9770f868e8c6a8cc042eb694bb9dd2ecee97160f6b5f63715217180e3943a840"
  }
}

Now, important thing to notice, is that the pieces layers attribute is not part of the info dictionary. Using the BEP9 - Extension for Peers to Send Metadata Files alone we will still be able to verify the integrity of each file after downloading its all pieces: in the file tree entry of the info dictionary, we have the pieces_root property which is the root of the Merkle Tree corresponding to that file. Unfortunately, we cannot verify the pieces as the info dictionary does not provide us with the hashes of the corresponding pieces. Even having access to the piece layer attribute from the torrent file, we would not be able to early verify integrity of the blocks we download.

BitTorrent version 2, solves that issue by introducing three new peer messages:

• hash request
• hashes
• hash reject

Later I will provide some examples of those messages, but here it is enough to say that with great flexibility, they allow us to retrieve the Merkle inclusion proofs for each file. Together with BEP9 - Extension for Peers to Send Metadata Files the peer can efficiently verify the contents being downloaded down to the 16 kiB level.

Extending Codex

From the above analysis, we see that for Codex protocol to be able to directly host BitTorrent content, we need the following extensions:

1. Peers need to advertise their corresponding SPRs under both info hash versions 1 and 2, so that they can be discovered.
2. To handle BitTorrent version 1 traffic, Codex peers need to be able to discover and download torrent metadata corresponding to the given info hash - in trackerless torrents, where the only input might be a magnet link containing only torrent's info hash, BitTorrent uses BEP9 - Extension for Peers to Send Metadata Files. In Codex, we will use BitTorrent Manifest to achieve the same. Read more: Implementing Codex BitTorrent extension.
3. We already use Merkle inclusion proofs, so we have great deal of flexibility of how we want to support torrents version 2. The info dictionary already provides us with the original file roots via pieces root, so we basically have to make sure we can provide the relevant intermediate layers aligned to the piece length (which basically means the leaves of our Merkle trees need to be computed over the chunks of the same size as in BitTorrent - 16 kiB) and the pieces root will be built on top of that. Moreover, having inclusion proofs in place we should be able to improve version 1 torrents as well. With original pieces hashes coming from the info dictionary, we can secure authenticity of the content, and with Codex inclusion proofs we can enhance torrents version 1 with early validation.

More detailed discussion will follow after learning more low level details of the Codex client.

Followup

Uploading and downloading content in Codex is were we document the contant upload and download in the Codex client.