From 9449bc1162dfa1987768c3ff3887fb5cdd09eeb0 Mon Sep 17 00:00:00 2001 From: Marcin Czenko Date: Fri, 21 Mar 2025 14:52:29 +0100 Subject: [PATCH] documents implementation of codex bittorrent extension --- ...Advertising BitTorrent content on Codex.md | 7 +- 10 Notes/BitTorrent Manifest.md | 19 + ...Implementing Codex BitTorrent extension.md | 573 ++++++++++++++++++ 10 Notes/Magnet Links.md | 36 ++ ...ad BitTorrent content with Codex - demo.md | 324 ++++++++++ .../92 Assets/BitTorrent-Upload-Download.svg | 3 + .../92 Assets/Codex BitTorrent Streaming.svg | 2 + 7 files changed, 961 insertions(+), 3 deletions(-) create mode 100644 10 Notes/BitTorrent Manifest.md create mode 100644 10 Notes/Implementing Codex BitTorrent extension.md create mode 100644 10 Notes/Magnet Links.md create mode 100644 10 Notes/Upload and download BitTorrent content with Codex - demo.md create mode 100644 90 Extras/92 Assets/BitTorrent-Upload-Download.svg create mode 100644 90 Extras/92 Assets/Codex BitTorrent Streaming.svg diff --git a/10 Notes/Advertising BitTorrent content on Codex.md b/10 Notes/Advertising BitTorrent content on Codex.md index 3b04a2d..5ef5bee 100644 --- a/10 Notes/Advertising BitTorrent content on Codex.md +++ b/10 Notes/Advertising BitTorrent content on Codex.md @@ -57,7 +57,7 @@ Now, let's assume our input is a magnet link corresponding to the above torrent magnet:?xt=urn:btih:1902d602db8c350f4f6d809ed01eff32f030da95&dn=data40k.bin ``` -The magnet link itself does not tell us details of the corresponding torrent file: what we have is only the hash of the bencoded `info` directory. How does the peer interested in downloading the content designated by the given magnet link finds the remaining details of the `info` dictionary? This is defined in [[BEP9 - Extension for Peers to Send Metadata Files]]. +The magnet link itself does not tell us details of the corresponding torrent file: what we have is only the hash of the b-encoded `info` directory. How does the peer interested in downloading the content designated by the given magnet link finds the remaining details of the `info` dictionary? This is defined in [[BEP9 - Extension for Peers to Send Metadata Files]]. > [!note] > BEP9 is also where the magnet links were introduced. @@ -314,11 +314,12 @@ Later I will provide some examples of those messages, but here it is enough to s From the above analysis, we see that for Codex protocol to be able to directly host BitTorrent content, we need the following extensions: 1. Peers need to advertise their corresponding SPRs under both `info` hash versions 1 and 2, so that they can be discovered. -2. To handle BitTorrent version 1 traffic, it is sufficient to support [[BEP9 - Extension for Peers to Send Metadata Files]]. +2. To handle BitTorrent version 1 traffic, Codex peers need to be able to discover and download torrent metadata corresponding to the given `info` hash - in trackerless torrents, where the only input might be a magnet link containing only torrent's `info` hash, BitTorrent uses [[BEP9 - Extension for Peers to Send Metadata Files]]. In Codex, we will use [[BitTorrent Manifest]] to achieve the same. Read more: [[Implementing Codex BitTorrent extension]]. 3. We already use Merkle inclusion proofs, so we have great deal of flexibility of how we want to support torrents version 2. The `info` dictionary already provides us with the original file roots via `pieces root`, so we basically have to make sure we can provide the relevant intermediate layers aligned to the piece length (which basically means the leaves of our Merkle trees need to be computed over the chunks of the same size as in BitTorrent - `16 kiB`) and the `pieces root` will be built on top of that. Moreover, having inclusion proofs in place we should be able to improve version 1 torrents as well. With original pieces hashes coming from the `info` dictionary, we can secure authenticity of the content, and with Codex inclusion proofs we can enhance torrents version 1 with early validation. More detailed discussion will follow after learning more low level details of the Codex client. ### Followup -[[Uploading and downloading content in Codex]] is were we document the contant upload and download in the Codex client. \ No newline at end of file +[[Uploading and downloading content in Codex]] is were we document the contant upload and download in the Codex client. + diff --git a/10 Notes/BitTorrent Manifest.md b/10 Notes/BitTorrent Manifest.md new file mode 100644 index 0000000..37dda64 --- /dev/null +++ b/10 Notes/BitTorrent Manifest.md @@ -0,0 +1,19 @@ +BitTorrent Manifest is a structure describing BitTorrent content and connecting it to the corresponding Codex content. + +It is defined as follows: + +```nim +type + BitTorrentPiece* = MultiHash + BitTorrentInfo* = ref object + length* {.serialize.}: uint64 + pieceLength* {.serialize.}: uint32 + pieces* {.serialize.}: seq[BitTorrentPiece] + name* {.serialize.}: ?string + + BitTorrentManifest* = ref object + info* {.serialize.}: BitTorrentInfo + codexManifestCid* {.serialize.}: Cid +``` + +It contains the BitTorrent `info` dictionary (here only version 1 of the protocol and no support for multiple files and dictionaries). diff --git a/10 Notes/Implementing Codex BitTorrent extension.md b/10 Notes/Implementing Codex BitTorrent extension.md new file mode 100644 index 0000000..8ba3047 --- /dev/null +++ b/10 Notes/Implementing Codex BitTorrent extension.md @@ -0,0 +1,573 @@ +In supporting BitTorrent on Codex network, it is important to clarify the pre-conditions: what do we expect to have as an input, and when will be the output. + +BitTorrent itself, can have three types of inputs: + +- a `.torrent` manifest file - a b-encoded [[BitTorrent metadata files]] - different formats for torrent version one and version 2 +- a magnet link - introduced in [[BEP9 - Extension for Peers to Send Metadata Files]] to support trackerless torrent and using DHT for peer discovery + +In both cases there are differences between version 1 and version 2 of metadata files (see [[BitTorrent metadata files]] for details) and version 1 and version 2 of [[Magnet Links|magnet links]]. + +A torrent file, provides a complete description of the torrent, and can be used to compute the corresponding `info` hash. + +Thus, while uploading (or seeding) BitTorrent content to the Codex network, the input is the content itself, while the output will be a (hybrid) magnet link. + +To retrieve previously seeded content, the input can be a torrent file, a magnet link, or directly an info hash (either v1 or v2, tagged or untagged). + +This is illustrated on the following picture: + +![[BitTorrent-Upload-Download.svg]] + +Thus, from the implementation perspective, the actual input to the Codex network while retrieving previously uploaded content is its `info` hash. + +### Uploading BitTorrent Content to Codex + +For the time being we only support version 1 and only a single file content (supporting directories and version 2 is work in progress). As a side not, limiting the description to this much simplified version will help to emphasize the important implementation challenges without being distracted with technicalities related to handling multiple file and folders. + +Thus, let's assume we have a single input file: `data40k.bin`. It is a binary file of size `40KiB` (`40960` Bytes). We will be using `16KiB` (`16384` Bytes) block size, and commonly used for such small content `piece length` of `256KiB` (`262144` Bytes). + +Let's go step by step through the code base to understand the upload process and the related challenges. + +First, the upload API: + +``` +/api/codex/v1/torrent +``` + +To upload the content we can use the following `POST` request: + +```bash +curl -X POST \ + http://localhost:8001/api/codex/v1/torrent \ + -H 'Content-Type: application/octet-stream' \ + -H 'Content-Disposition: filename="data40k.bin"' \ + -w '\n' \ + -T data40k.bin +``` + +We use `Content Disposition` header to indicate the name we want to use for the uploaded content. + +This will land to the API handler in `codex/rest/api.nim` : + +```nim +router.rawApi(MethodPost, "/api/codex/v1/torrent") do() -> RestApiResponse: + ## Upload a file in a streaming manner + ## +``` + +It will call `node.storeTorrent` to effectively upload the content and get the resulting `info` (multi) hash back: + +```nim +without infoHash =? ( + await node.storeTorrent( + AsyncStreamWrapper.new(reader = AsyncStreamReader(reader)), + filename = filename, + mimetype = mimetype, + ) +), error: + error "Error uploading file", exc = error.msg + return RestApiResponse.error(Http500, error.msg) +``` + +This brings us to `node.storeTorrent` in `codex/node.nim: + +```nim +proc storeTorrent*( + self: CodexNodeRef, + stream: LPStream, + filename: ?string = string.none, + mimetype: ?string = string.none, +): Future[?!MultiHash] {.async.} = + info "Storing BitTorrent data" + + without bitTorrentManifest =? + await self.storePieces( + stream, filename = filename, mimetype = mimetype, blockSize = BitTorrentBlockSize + ): + return failure("Unable to store BitTorrent data") + + trace "Created BitTorrent manifest", bitTorrentManifest = $bitTorrentManifest + + let infoBencoded = bencode(bitTorrentManifest.info) + + trace "BitTorrent Info successfully bencoded" + + without infoHash =? MultiHash.digest($Sha1HashCodec, infoBencoded).mapFailure, err: + return failure(err) + + trace "computed info hash", infoHash = $infoHash + + without manifestBlk =? await self.storeBitTorrentManifest( + bitTorrentManifest, infoHash + ), err: + error "Unable to store manifest" + return failure(err) + + info "Stored BitTorrent data", + infoHash = $infoHash, codexManifestCid = bitTorrentManifest.codexManifestCid + + success infoHash +``` + +It starts with `self.storePieces`, which returns a [[BitTorrent Manifest]]. A manifest contains the BitTorrent Info dictionary and the corresponding Codex Manifest Cid: + +``` +type + BitTorrentInfo* = ref object + length*: uint64 + pieceLength*: uint32 + pieces*: seq[MultiHash] + name*: ?string + + BitTorrentManifest* = ref object + info*: BitTorrentInfo + codexManifestCid*: Cid +``` + +`storePieces` does a very similar job to the `store` proc which is used for the regular Codex content, but additionally, it computes the *piece hashes* and creates the `info` dictionary and finally returns the corresponding `BitTorrentManifest`. + +Back in `storeTorrent`, we *b-encode* the `info` dictionary and compute its hash (multihash). This `info` (multi) hash is what we will use to announce the content on the Codex DHT (see [[Announcing BitTorrent Content on Codex DHT]]). + +Finally, `storeBitTorrentManifest` will effectively store the BitTorrent manifest block on the Codex network: + +``` +proc storeBitTorrentManifest*( + self: CodexNodeRef, manifest: BitTorrentManifest, infoHash: MultiHash +): Future[?!bt.Block] {.async.} = + let encodedManifest = manifest.encode() + + without infoHashCid =? Cid.init(CIDv1, InfoHashV1Codec, infoHash).mapFailure, error: + trace "Unable to create CID for BitTorrent info hash" + return failure(error) + + without blk =? bt.Block.new(data = encodedManifest, cid = infoHashCid, verify = false), + error: + trace "Unable to create block from manifest" + return failure(error) + + if err =? (await self.networkStore.putBlock(blk)).errorOption: + trace "Unable to store BitTorrent manifest block", cid = blk.cid, err = err.msg + return failure(err) + + success blk +``` + +Some important things happen here. First, notice, that in Codex we use `Cids` to refer to the content. This is very handy: requesting a cid and getting the corresponding data back, we can immediately check if the content multihash present in the Cid, matches the computed multihash of the received data. If they do not match, we know immediately that the received block is invalid. + +But looking at the code above, a careful reader will spot immediately that we are *cheating* a bit. + +We first create a cid (`infoHashCid`) using precomputed `infoHash`, which we then associate with the `encodedManifest` in the `Block.new` call. Clearly, `info` hash does not identify our `encodedManifest`: if we compute a hash of the `encodedManifest`, it would not match the precomputed `infoHash`. This is because our Bit Torrent Manifest is more than just the `info` dictionary: it also contains the cid of the corresponding Codex Manifest for our content. + +This cid is clearly not a valid cid. + +We could create a valid Cid, by, for instance, creating a hash over the whole `encodedManifest` and appending it to the precomputed `infoHash` in such a Cid. Then, while retrieving the corresponding block back, we could first compare that the computed hash over the retrieved data matches the hash of the `encodedManifest` that we included in our cid, and then after reconstructing the BitTorrent Manifest from the encoded data, we could b-encode the `info` dictionary from the reconstructed BitTorrent Manifest, compute its hash, and compare it with the precomputed `infoHash` included in the cid. This would make the cid valid, but there is a problem with this approach. + +In Codex, we use cids as references to blocks in `RepoStore`. We namely use cids as inputs to functions like `createBlockExpirationMetadataKey` or `makePrefixKey`. The cid itself is not preserved. The uploader (the seeder) has all necessary data to create an extended cid we describe in the paragraph above, but when requesting, the downloader knows only the `info` hash or potentially the contents of the the `.torrent` metadata file. In any case, the downloader does not know the cid of the underlying Codex manifest, pointing to the actual data. This means that the downloader is unable to create a full cid with the appended hash of the full `encodedManifest`. It is technically possible to send such an incomplete cid and use it to retrieve the full cid from the uploader datastore, but we are not making the system any more secure by doing this. The sender, can easily send a forged block with with perfectly valid cid as it has all necessary information to compute the appended hash, but the receiver, not having access to this information beforehand, will not be able to validate it. + +Does it mean we can only be sure that the received content identified by the cid of the Codex manifest matches the requested info hash? No. + +Notice, that BitTorrent does not use cids. The BitTorrent protocol operates at the level of pieces, and in version 1 of the protocol does not even use inclusion proofs. Yet, it does not wait till the whole piece is fetched in order to conclude it is genuine. + +The info dictionary contains the `pieces` attribute, with hashes for all pieces. Once the piece is aggregated from the underlying blocks of `16KiB`, the hash is computed and compared against an entry in the `pieces` array. And this exactly what we do in Codex in order to prove that the received data, identified by the cid of the Codex manifest, matches the requested `info` hash. +Moreover, we also validate the received data at the block level, even before being able to validate the complete piece. We get this as a bonus from the Codex protocol, which together with data block, sends also the corresponding inclusion proof. Thus, even though at the moment we validate the individual blocks, we do not know if the received data, identified by the cid of the Codex manifest, matches the requested `info` hash, we do know already if the received data matches the Codex manifest. If this is not the case, if does not even make sense to aggregate pieces. + +Thus, to summarize, while we cannot validate if the received BitTorrent manifest points to the valid data by validating the corresponding cid (`infoHashCid`), we do it later in two phases. Let's look at the download flow, starting from the end. + +### Downloading BitTorrent Content from Codex + +We start from the `NetworkPeer.readLoop` (in `codex/blockexchange/network/networkpeer.nim`), where we decode the protocol `Message` with: + +```nim +data = await conn.readLp(MaxMessageSize.int) +msg = Message.protobufDecode(data).mapFailure().tryGet() +``` + +There, for each data item, we call: + +```nim +BlockDelivery.decode(initProtoBuffer(item, maxSize = MaxBlockSize)) +``` + +and this is where we get the cid, `Block`, `BlockAddress`, and the corresponding `proof` (for regular data, or *leaf* blocks): + +```nim +proc decode*(_: type BlockDelivery, pb: ProtoBuffer): ProtoResult[BlockDelivery] = + var + value = BlockDelivery() + dataBuf = newSeq[byte]() + cidBuf = newSeq[byte]() + cid: Cid + ipb: ProtoBuffer + + if ?pb.getField(1, cidBuf): + cid = ?Cid.init(cidBuf).mapErr(x => ProtoError.IncorrectBlob) + if ?pb.getField(2, dataBuf): + value.blk = + ?Block.new(cid, dataBuf, verify = true).mapErr(x => ProtoError.IncorrectBlob) + if ?pb.getField(3, ipb): + value.address = ?BlockAddress.decode(ipb) + + if value.address.leaf: + var proofBuf = newSeq[byte]() + if ?pb.getField(4, proofBuf): + let proof = ?CodexProof.decode(proofBuf).mapErr(x => ProtoError.IncorrectBlob) + value.proof = proof.some + else: + value.proof = CodexProof.none + else: + value.proof = CodexProof.none + + ok(value) +``` + +We see that we while constructing instance of `Block`, we already request the block validation by setting `verify = true`: + +```nim +proc new*( + T: type Block, cid: Cid, data: openArray[byte], verify: bool = true +): ?!Block = + ## creates a new block for both storage and network IO + ## + + without isTorrent =? cid.isTorrentCid, err: + return "Unable to determine if cid is torrent info hash".failure + + # info hash cids are "fake cids" - they will not validate + # info hash validation is done outside of the cid itself + if verify and not isTorrent: + let + mhash = ?cid.mhash.mapFailure + computedMhash = ?MultiHash.digest($mhash.mcodec, data).mapFailure + computedCid = ?Cid.init(cid.cidver, cid.mcodec, computedMhash).mapFailure + if computedCid != cid: + return "Cid doesn't match the data".failure + + return Block(cid: cid, data: @data).success +``` + +Here we see that because as explained above, the cids corresponding to the BitTorrent manifest blocks cannot be immediately validated, we make sure, the validation is skipped here for Torrent cids. + +Once the `Message` is decoded, back in `NetworkPeer.readLoop`, it is passed to `NetworkPeer.handler` which is set to `Network.rpcHandler` while creating the instance of `NetworkPeer` in `Network.getOrCreatePeer`. For block deliveries, `Network.rpcHandler` forwards `msg.payload` (`seq[BlockDelivery]`) to `Network.handleBlocksDelivery`, which in turn, calls `Network.handlers.onBlocksDelivery`. The `Network.handlers.onBlocksDelivery` is set by the constructor of `BlockExcEngine`. Thus, in the end of its journey, a `seq[BlockDelivery]` from the `msg.payload` ends up in `BlockExcEngine.blocksDeliveryHandler`. This is where the data blocks are further validated against the inclusion proof and then the validated data (*leafs*) blocks or non-data blocks (non-*leafs*, e.g. a BitTorrent or Codex Manifest block), are stored in the `localStore` and then *resolved* against pending blocks via `BlockExcEngine.resolveBlocks` that calls `pendingBlocks.resolve(blocksDelivery)` (`PendingBlocksManager`). This is where `blockReq.handle.complete(bd.blk)` is called on the matching pending blocks, which completes future awaited in `BlockExcEngine.requestBlock`, which completes the future awaited in `NetworkStore.getBlock`: `await self.engine.requestBlock(address)`. And `NetworkStore.getBlock` was awaited either in `CodexNodeRef.fetchPieces` for data blocks or in `CodexNodeRef.fetchTorrentManifest`. + +So, how do we get to `CodexNodeRef.fetchPieces` and `CodexNodeRef.fetchTorrentManifest` in the download flow. + +It starts with the API handler of `/api/codex/v1/torrent/{infoHash}/network/stream`: + +```nim +router.api(MethodGet, "/api/codex/v1/torrent/{infoHash}/network/stream") do( + infoHash: MultiHash, resp: HttpResponseRef + ) -> RestApiResponse: + var headers = buildCorsHeaders("GET", allowedOrigin) + + without infoHash =? infoHash.mapFailure, error: + return RestApiResponse.error(Http400, error.msg, headers = headers) + + if infoHash.mcodec != Sha1HashCodec: + return RestApiResponse.error( + Http400, "Only torrents version 1 are currently supported!", headers = headers + ) + + if corsOrigin =? allowedOrigin: + resp.setCorsHeaders("GET", corsOrigin) + resp.setHeader("Access-Control-Headers", "X-Requested-With") + + trace "torrent requested: ", multihash = $infoHash + + await node.retrieveInfoHash(infoHash, resp = resp) +``` + +`CodexNodeRef.retrieveInfoHash` first tries to fetch the `Torrent` object, which consists of `torrentManifest` and `codexManifest`. To get it, it calls `node.retrieveTorrent(infoHash)` with the `infoHash` as the argument. And then in the `retrieveTorrent` we get to the above mentioned `fetchTorrentManifest`: + +```nim +proc retrieveTorrent*( + self: CodexNodeRef, infoHash: MultiHash +): Future[?!Torrent] {.async.} = + without infoHashCid =? Cid.init(CIDv1, InfoHashV1Codec, infoHash).mapFailure, error: + trace "Unable to create CID for BitTorrent info hash" + return failure(error) + + without torrentManifest =? (await self.fetchTorrentManifest(infoHashCid)), err: + trace "Unable to fetch Torrent Manifest" + return failure(err) + + without codexManifest =? (await self.fetchManifest(torrentManifest.codexManifestCid)), + err: + trace "Unable to fetch Codex Manifest for torrent info hash" + return failure(err) + + success (torrentManifest: torrentManifest, codexManifest: codexManifest) +``` + +We first create `infoHashCid`, using only the precomputed `infoHash` and we pass it to `fetchTorrentManifest`: + +```nim +proc fetchTorrentManifest*( + self: CodexNodeRef, infoHashCid: Cid +): Future[?!BitTorrentManifest] {.async.} = + if err =? infoHashCid.isTorrentInfoHash.errorOption: + return failure "CID has invalid content type for torrent info hash {$cid}" + + trace "Retrieving torrent manifest for infoHashCid", infoHashCid + + without blk =? await self.networkStore.getBlock(BlockAddress.init(infoHashCid)), err: + trace "Error retrieve manifest block", infoHashCid, err = err.msg + return failure err + + trace "Successfully retrieved torrent manifest with given block cid", + cid = blk.cid, infoHashCid + trace "Decoding torrent manifest" + + without torrentManifest =? BitTorrentManifest.decode(blk), err: + trace "Unable to decode torrent manifest", err = err.msg + return failure("Unable to decode torrent manifest") + + trace "Decoded torrent manifest", infoHashCid, torrentManifest = $torrentManifest + + without isValid =? torrentManifest.validate(infoHashCid), err: + trace "Error validating torrent manifest", infoHashCid, err = err.msg + return failure(err.msg) + + if not isValid: + trace "Torrent manifest does not match torrent info hash", infoHashCid + return failure "Torrent manifest does not match torrent info hash {$infoHashCid}" + + return torrentManifest.success +``` + +Here we will be awaiting for the `networkStore.getBlock`, which will get completed with the block delivery flow we describe at the beginning of this section. We restore the `BitTorrentManifest` object using `BitTorrentManifest.decode(blk)`, and then we validate if the `info` dictionary from the received BitTorrent manifest matches the request `infoHash`: + +```nim +without isValid =? torrentManifest.validate(infoHashCid), err: + trace "Error validating torrent manifest", infoHashCid, err = err.msg + return failure(err.msg) +``` + +With this we know that we have a genuine `info` dictionary. + +Now, we still need to get and validate the actual data. + +BitTorrent manifest includes the cid of the Codex manifest in `codexManifestCid` attribute. Back in `retrieveTorrent`, we now fetch the Codex manifest, and we return both to `retrieveInfoHash`, where the download effectively started. + +We are ready to start the streaming operation. The illustration below, shows a high level conceptual overview of the streaming process: + +![[Codex BitTorrent Streaming.svg]] + +From the diagram above we see, that there are two concurrent tasks: a *prefetch* tasks fetching the blocks from the network store, aggregating, and validating pieces, and a *streaming* tasks, sending the blocks down to the client via REST API. + +The prefetch task is started by `retrieveInfoHash` calling `streamTorrent`, passing both manifests and piece validator as arguments: + +```nim +let torrentPieceValidator = newTorrentPieceValidator(torrentManifest, codexManifest) + +let stream = + await node.streamTorrent(torrentManifest, codexManifest, torrentPieceValidator) +``` + +Let's take a look at `streamTorrent`: + +```nim +proc streamTorrent*( + self: CodexNodeRef, + torrentManifest: BitTorrentManifest, + codexManifest: Manifest, + pieceValidator: TorrentPieceValidator, +): Future[LPStream] {.async: (raises: []).} = + trace "Retrieving pieces from torrent" + let stream = LPStream(StoreStream.new(self.networkStore, codexManifest, pad = false)) + + proc prefetch(): Future[void] {.async: (raises: []).} = + try: + if err =? (await self.fetchPieces(torrentManifest, codexManifest, pieceValidator)).errorOption: + error "Unable to fetch blocks", err = err.msg + await noCancel pieceValidator.cancel() + await noCancel stream.close() + except CancelledError: + trace "Prefetch cancelled" + + let prefetchTask = prefetch() + + # Monitor stream completion and cancel background jobs when done + proc monitorStream() {.async: (raises: []).} = + try: + await stream.join() + except CancelledError: + trace "Stream cancelled" + finally: + await noCancel prefetchTask.cancelAndWait + + self.trackedFutures.track(monitorStream()) + + trace "Creating store stream for torrent manifest" + stream +``` + +`streamTorrent` does two things: + +1. starts background `prefetch` task +2. monitors the stream using `monitorStream` + +The `prefetch` job calls `fetchPieces`: + +```nim +proc fetchPieces*( + self: CodexNodeRef, + torrentManifest: BitTorrentManifest, + codexManifest: Manifest, + pieceValidator: TorrentPieceValidator, +): Future[?!void] {.async: (raises: [CancelledError]).} = + let cid = codexManifest.treeCid + let numOfBlocksPerPiece = pieceValidator.numberOfBlocksPerPiece + let blockIter = Iter[int].new(0 ..< codexManifest.blocksCount) + while not blockIter.finished: + let blockFutures = collect: + for i in 0 ..< numOfBlocksPerPiece: + if not blockIter.finished: + let address = BlockAddress.init(cid, blockIter.next()) + self.networkStore.getBlock(address) + + without blocks =? await allFinishedValues(blockFutures), err: + return failure(err) + + if err =? self.validatePiece(pieceValidator, blocks).errorOption: + return failure(err) + + await sleepAsync(1.millis) + + success() +``` + +We fetch blocks in *batches*, or rather *in pieces*. We trigger fetching blocks with `self.networkStore.getBlock(address)`, which will resolve by either getting the block from the local store or from the network through block delivery described earlier. + +Notice that we need to get all the relevant blocks here, not only the blocks that are not yet in the local store. This is necessary, because we need to get all the blocks in a piece so that we can validate the piece and potentially stop streaming if the piece turns out to be invalid. + +Before calling `validatePiece`, where validation takes place, we wait for all `Futures` to complete returning the requested `blocks`. + +`validatePiece` is defined as follows: + +```nim +proc validatePiece( + self: CodexNodeRef, pieceValidator: TorrentPieceValidator, blocks: seq[bt.Block] +): ?!void {.raises: [].} = + trace "Fetched complete torrent piece - verifying..." + let pieceIndex = pieceValidator.validatePiece(blocks) + + if pieceIndex < 0: + error "Piece verification failed", pieceIndex = pieceIndex + return failure(fmt"Piece verification failed for {pieceIndex=}") + + trace "Piece successfully verified", pieceIndex + + let confirmedPieceIndex = pieceValidator.confirmCurrentPiece() + + if pieceIndex != confirmedPieceIndex: + error "Piece confirmation failed", + pieceIndex = pieceIndex, confirmedPieceIndex = confirmedPieceIndex + return + failure(fmt"Piece confirmation failed for {pieceIndex=}, {confirmedPieceIndex=}") + success() +``` + +It first calls `validatePiece` on the `pieceValidator`, which computes the SHA1 hash of the concatenated blocks and checks if it matches the (multi) hash from the `info` dictionary. + +> [!info] +This constitutes the second validation step: after we checked that the `info` dictionary matches the requested `info` hash in the first step described above, here we are making sure that the received content matches the metadata in the `info` dictionary, and thus it is indeed the content identified by the `info` hash from the request. + +`PiecePalidator` maintains internal state so that it known which piece is expected at the given moment - this is why it does not need the piece index argument to validate the blocks. Upon successful validation it returns the index of the validated piece. We then call `pieceValidator.confirmCurrentPiece` to *notify* REST API streaming that is awaiting on `torrentPieceValidator.waitForNextPiece()` before streaming the validated blocks to the requesting client: + +```nim +proc retrieveInfoHash( + node: CodexNodeRef, infoHash: MultiHash, resp: HttpResponseRef +): Future[void] {.async.} = + ## Download torrent from the node in a streaming + ## manner + ## + var stream: LPStream + + var bytes = 0 + try: + without torrent =? (await node.retrieveTorrent(infoHash)), err: + error "Unable to fetch Torrent Metadata", err = err.msg + resp.status = Http404 + await resp.sendBody(err.msg) + return + let (torrentManifest, codexManifest) = torrent + + if codexManifest.mimetype.isSome: + resp.setHeader("Content-Type", codexManifest.mimetype.get()) + else: + resp.addHeader("Content-Type", "application/octet-stream") + + if codexManifest.filename.isSome: + resp.setHeader( + "Content-Disposition", + "attachment; filename=\"" & codexManifest.filename.get() & "\"", + ) + else: + resp.setHeader("Content-Disposition", "attachment") + + await resp.prepareChunked() + + let torrentPieceValidator = newTorrentPieceValidator(torrentManifest, codexManifest) + + let stream = + await node.streamTorrent(torrentManifest, codexManifest, torrentPieceValidator) + + while not stream.atEof: + trace "Waiting for piece..." + let pieceIndex = await torrentPieceValidator.waitForNextPiece() + + if -1 == pieceIndex: + warn "No more torrent pieces expected. TorrentPieceValidator might be out of sync!" + break + + trace "Got piece", pieceIndex + + let blocksPerPieceIter = torrentPieceValidator.getNewBlocksPerPieceIterator() + while not blocksPerPieceIter.finished and not stream.atEof: + var buff = newSeqUninitialized[byte](BitTorrentBlockSize.int) + # wait for the next the piece to prefetch + let len = await stream.readOnce(addr buff[0], buff.len) + + buff.setLen(len) + if buff.len <= 0: + break + + bytes += buff.len + + await resp.sendChunk(addr buff[0], buff.len) + discard blocksPerPieceIter.next() + await resp.finish() + codex_api_downloads.inc() + except CancelledError as exc: + info "Stream cancelled", exc = exc.msg + raise exc + except CatchableError as exc: + warn "Error streaming blocks", exc = exc.msg + resp.status = Http500 + if resp.isPending(): + await resp.sendBody(exc.msg) + finally: + info "Sent bytes for torrent", infoHash = $infoHash, bytes + if not stream.isNil: + await stream.close() +``` + +Now, two important points. First, when the streaming happens to be interrupted the stream will be closed in the `finally` block. This in turns will be detected by the `monitorStream` in `streamTorrent` causing the `prefetch` job to be cancelled. Second, when either piece validation fails, or if any of the `getBlock` future awaiting completion fails, `prefetch` will return error, which will cause the stream to be closed: + +```nim +proc prefetch(): Future[void] {.async: (raises: []).} = + try: + if err =? ( + await self.fetchPieces(torrentManifest, codexManifest, onPieceReceived) + ).errorOption: + error "Unable to fetch blocks", err = err.msg + await noCancel pieceValidator.cancel() + await noCancel stream.close() + except CancelledError: + trace "Prefetch cancelled" +``` + +Without this detection mechanism, we would either continue fetching blocks even when streaming API request has been interrupted, or we would continue streaming, even when it is already known that the piece validation phase has failed. This would result in potentially invalid content being returned to the client. After any failure in the `prefetch` job, the pieces will no longer be validated, and thus it does not make any sense to continue the streaming operation. + +The `stream.readOnce` called in the streaming loop and implemented in `StoreStream`, which uses the same underlying `networkStore` that is also used in `fetchPieces` proc shown above, will be calling that same `getBlock` operation, which in case the block is not already in local store (because it was already there or as a result of the prefetch operation), will request it from the block exchange engine via `BlockExcEngine.requestBlock` operation. In case there is already a pending request for the given block address, the `PendingBlocksManager` will return the existing block handle, so that `BlockExcEngine.requestBlock` operation will not cause duplicate request. It will, however potentially return an invalid block to the client, before the containing piece has been validated in the prefetch phase. + +If you want to experiment with uploading and downloading BitTorrent content yourself, check [[Upload and download BitTorrent content with Codex - demo]]. \ No newline at end of file diff --git a/10 Notes/Magnet Links.md b/10 Notes/Magnet Links.md new file mode 100644 index 0000000..24c6154 --- /dev/null +++ b/10 Notes/Magnet Links.md @@ -0,0 +1,36 @@ +Magnet links has been introduced in [[BEP9 - Extension for Peers to Send Metadata Files]]. Here is a short summary with examples. + +Magnet link format: + +``` +v1: magnet:?xt=urn:btih:&dn=&tr=&x.pe= +v2: magnet:?xt=urn:btmh:&dn=&tr=&x.pe= +``` + +BEP9 provides detailed description of all the fields. Here it is important to emphasize that only `xt` is mandatory. In particular for our purpose, in trackerless torrents, we may see in practice the magnet links having only the `xt` attribute. Other attributes, especially `x.pe` (peer address), may be used as well, but our discovery should not relay on any other attributes than `xt`. + +Tow examples of such a simplest magnet-links: + +**Version 1** + +``` +magnet:?xt=urn:btih:1902d602db8c350f4f6d809ed01eff32f030da95 +``` + +**Version 2** + +``` +magnet:?xt=urn:btmh:122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1 +``` + +**Hybrid** + +``` +magnet:?xt=urn:btmh:122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1&urn:btih:1902d602db8c350f4f6d809ed01eff32f030da95 +``` + +`122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1` is a SHA-256 multihash: + +``` +sha2-256/03ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1 +``` diff --git a/10 Notes/Upload and download BitTorrent content with Codex - demo.md b/10 Notes/Upload and download BitTorrent content with Codex - demo.md new file mode 100644 index 0000000..85a5239 --- /dev/null +++ b/10 Notes/Upload and download BitTorrent content with Codex - demo.md @@ -0,0 +1,324 @@ +For the cleaner and more focused logging, it is adviced to compile the codex client using the following options in `build.nims`: + +```nim +task codex, "build codex binary": + buildBinary "codex", + # default params + # params = "-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE" + params = + "-d:chronicles_runtime_filtering -d:chronicles_log_level=TRACE -d:chronicles_enabled_topics:restapi:TRACE,node:TRACE" +``` + +In the example session, we will use two nodes: + +- node-1 will be where we will upload (seed) the content +- node-2 will be from where we will be downloading the previously uploaded content. + +Start node-1: + +```bash +./build/codex --data-dir=./data-1 --listen-addrs=/ip4/127.0.0.1/tcp/8081 --api-port=8001 --nat=none --disc-port=8091 --log-level=TRACE +``` + + +Generate fresh content: + +```bash +dd if=/dev/urandom of=./data10M.bin bs=10M count=1 +``` + +Upload content to node-1: + +```bash +curl -X POST \ + http://localhost:8001/api/codex/v1/torrent \ + -H 'Content-Type: application/octet-stream' \ + -H 'Content-Disposition: filename="data10M.bin"' \ + -w '\n' \ + -T data10M.bin +11144249FFB943675890CF09342629CD3782D107B709 +``` + +Set `info_hash` env var: + +```bash +export INFO_HASH=11144249FFB943675890CF09342629CD3782D107B709 +``` + +Start node-2: + +```bash +./build/codex --data-dir=./data-2 --listen-addrs=/ip4/127.0.0.1/tcp/8082 --api-port=8002 --nat=none --disc-port=8092 --log-level=TRACE +``` + +Make sure that node-1 and node-2 are connect (should be automatic but sometimes we need to trigger it when running on localhost without nat). + +Get the peerId from node-1 + +```bash +curl -H 'Accept: text/plain' 'http://localhost:8001/api/codex/v1/peerid' --write-out '\n' +16Uiu2HAmSfUAJfzaMadXynZCeVza4gFMR6krZM4mnZjYU6tQ2jog +``` + +Here it is also good to use env var: + +```bash +export PEERID_NODE1=$(curl -H 'Accept: text/plain' 'http://localhost:8001/api/codex/v1/peerid') +``` + +Connect node-2 to node-1: + +```bash +curl "http://localhost:8002/api/codex/v1/connect/${PEERID_NODE1}?addrs=/ip4/127.0.0.1/tcp/8081" +Successfully connected to peer +``` + +Make sure `INFO_HASH` env var is set: + +```bash +export INFO_HASH=11144249FFB943675890CF09342629CD3782D107B709 +``` + +Stream the content from node-2: + +```bash +curl "http://localhost:8002/api/codex/v1/torrent/${INFO_HASH}/network/stream" -o "${INFO_HASH}.bin" +``` + +And to get just the torrent manifest: + +```bash +curl "http://localhost:8002/api/codex/v1/torrent/${INFO_HASH}/network/manifest" | jq + % Total % Received % Xferd Average Speed Time Time Time Current + Dload Upload Total Spent Left Speed +100 2113 100 2113 0 0 853k 0 --:--:-- --:--:-- --:--:-- 1031k +{ + "infoHash": "111404010165844CF71179B42AC3C0462E31EB9E74B1", + "torrentManifest": { + "info": { + "length": 10485760, + "pieceLength": 262144, + "pieces": [ + "111494BFE9E914669431DAF5446732E399EF3533820F", + "1114DFB0389F65DBB285B3FB21B057408B5CAC27A188", + "111417798F99E9CC9F6C0A96C00C667ED89A8187CEE9", + "11146CFE9BFE2901BBA9DE573A0FE3613F77E26053FD", + "111447E8129FEACB16613C77F687F7ABE88A7D3AF34F", + "11140051FC4683B1E98CA24AD32ED05C0B6A02A9CE35", + "11149C67E50A767DF3F910FB872C887B560B9E59BC64", + "11141AC926FE56830A1DF069F979E4B2E6EE8E80E02B", + "11146E372C5814D5EAEC958ACF15A09A25DDD254FD5F", + "1114A494C6185B2C6697898F2B7519357DAAB52CCE4E", + "11148FE075ED85EE50D056DB82A4C6802AC145AB14C6", + "111479745CDC0A973D19A86EF8A7CA6BBF0A066183DE", + "1114EB3C71D98ED8C51D8705A3562FFC816787B2BB63", + "11146C4486F224B217BA6C7DDB41874631862AA987B3", + "1114E32FFD30458E384E4B5C4311EC6CD60806B9E6E2", + "1114698D13326C5DF59A95345D1FC5FEF4EE50A27EA1", + "11147446B0137417EDBDE1EA4CAC5DB8F9A8DD30DDAE", + "11149C1F5C49EB23134ECBB1007C3964F9D6DBC601A0", + "1114656F38874EDD3E7D18949C1F716FECFE0DAE6D48", + "1114AFDB1ED7DBA0DA42567F369B4D009D2917803711", + "1114384092051DBBBE7BA04815925941C8674F2FD5A4", + "111453026DA2899EB0AD8E7285B8CE2FD837513CEEBE", + "1114F4A106192FE40C6ECF94D77DE62297AC3B4CFBC2", + "1114E7D6C4633AFA8966819358179531352E35587807", + "1114D16F9822A32589C3AFBD73A51819B3261AAD8E67", + "1114985F1B5C2420F9984D285D2884B999B53031B462", + "11145DE360A86D1105951F346F48350720A3E665A800", + "111400E9988CACF7F0C637B2AE2CFD0A291FF7FD05C8", + "1114D2EE427BA371FC157F30BC3467C87AC00581372B", + "1114F00AE60ED70619C1829BF2E8261226D0467F175C", + "111450650C77BACCE80C8833A1FCE3565D7252BDA8E2", + "11141CE38B812A5A07AAEB99878C207146CA8181EC55", + "11142503901ED89ED51E927873D39F5DCDD44F454DF2", + "11143FBB54D2ECB2AB33953F090DF33279BBE20FC5B8", + "1114048E4CE97C3DB74BD5F680F87E58FBCCA9A54EDD", + "11149CF6AF5F6E71EAB600A4E156F27C189651D6C8D2", + "1114467578C62EED05FDCF29BABD3CD6989709B3E419", + "1114183914076A0BF45133D56FBFF9FC510AA76E4462", + "11141012A7A054710B316DBA414EED131A17508CCC96", + "1114D680DBBC01022878727D41FB86A1954474026139" + ], + "name": "data10M.bin" + }, + "codexManifestCid": "zDvZRwzm5J2n6S12QWCHm3PdiavyQtLbC73txVr1DQoRUhyLKyUA" + } +} +``` + +And here is the streaming log (node-2) for the reference: + +```bash +TRC 2025-03-20 03:32:28.939+01:00 torrent requested: topics="codex restapi" tid=6977524 multihash=sha1/04010165844CF71179B42AC3C0462E31EB9E74B1 +TRC 2025-03-20 03:32:28.940+01:00 Retrieving torrent manifest for infoHashCid topics="codex node" tid=6977524 infoHashCid=z8q*sJUUWC +TRC 2025-03-20 03:32:28.946+01:00 Successfully retrieved torrent manifest with given block cid topics="codex node" tid=6977524 cid=z8q*sJUUWC infoHashCid=z8q*sJUUWC +TRC 2025-03-20 03:32:28.946+01:00 Decoding torrent manifest topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:28.946+01:00 Decoded torrent manifest topics="codex node" tid=6977524 infoHashCid=z8q*sJUUWC torrentManifest="BitTorrentManifest(info: BitTorrentInfo(length: 10485760, pieceLength: 262144, pieces: @[sha1/94BFE9E914669431DAF5446732E399EF3533820F, sha1/DFB0389F65DBB285B3FB21B057408B5CAC27A188, sha1/17798F99E9CC9F6C0A96C00C667ED89A8187CEE9, sha1/6CFE9BFE2901BBA9DE573A0FE3613F77E26053FD, sha1/47E8129FEACB16613C77F687F7ABE88A7D3AF34F, sha1/0051FC4683B1E98CA24AD32ED05C0B6A02A9CE35, sha1/9C67E50A767DF3F910FB872C887B560B9E59BC64, sha1/1AC926FE56830A1DF069F979E4B2E6EE8E80E02B, sha1/6E372C5814D5EAEC958ACF15A09A25DDD254FD5F, sha1/A494C6185B2C6697898F2B7519357DAAB52CCE4E, sha1/8FE075ED85EE50D056DB82A4C6802AC145AB14C6, sha1/79745CDC0A973D19A86EF8A7CA6BBF0A066183DE, sha1/EB3C71D98ED8C51D8705A3562FFC816787B2BB63, sha1/6C4486F224B217BA6C7DDB41874631862AA987B3, sha1/E32FFD30458E384E4B5C4311EC6CD60806B9E6E2, sha1/698D13326C5DF59A95345D1FC5FEF4EE50A27EA1, sha1/7446B0137417EDBDE1EA4CAC5DB8F9A8DD30DDAE, sha1/9C1F5C49EB23134ECBB1007C3964F9D6DBC601A0, sha1/656F38874EDD3E7D18949C1F716FECFE0DAE6D48, sha1/AFDB1ED7DBA0DA42567F369B4D009D2917803711, sha1/384092051DBBBE7BA04815925941C8674F2FD5A4, sha1/53026DA2899EB0AD8E7285B8CE2FD837513CEEBE, sha1/F4A106192FE40C6ECF94D77DE62297AC3B4CFBC2, sha1/E7D6C4633AFA8966819358179531352E35587807, sha1/D16F9822A32589C3AFBD73A51819B3261AAD8E67, sha1/985F1B5C2420F9984D285D2884B999B53031B462, sha1/5DE360A86D1105951F346F48350720A3E665A800, sha1/00E9988CACF7F0C637B2AE2CFD0A291FF7FD05C8, sha1/D2EE427BA371FC157F30BC3467C87AC00581372B, sha1/F00AE60ED70619C1829BF2E8261226D0467F175C, sha1/50650C77BACCE80C8833A1FCE3565D7252BDA8E2, sha1/1CE38B812A5A07AAEB99878C207146CA8181EC55, sha1/2503901ED89ED51E927873D39F5DCDD44F454DF2, sha1/3FBB54D2ECB2AB33953F090DF33279BBE20FC5B8, sha1/048E4CE97C3DB74BD5F680F87E58FBCCA9A54EDD, sha1/9CF6AF5F6E71EAB600A4E156F27C189651D6C8D2, sha1/467578C62EED05FDCF29BABD3CD6989709B3E419, sha1/183914076A0BF45133D56FBFF9FC510AA76E4462, sha1/1012A7A054710B316DBA414EED131A17508CCC96, sha1/D680DBBC01022878727D41FB86A1954474026139], name: some(\"data10M.bin\")), codexManifestCid: zDvZRwzm5J2n6S12QWCHm3PdiavyQtLbC73txVr1DQoRUhyLKyUA)" +TRC 2025-03-20 03:32:28.946+01:00 Retrieving manifest for cid topics="codex node" tid=6977524 cid=zDv*yLKyUA +TRC 2025-03-20 03:32:28.949+01:00 Decoding manifest for cid topics="codex node" tid=6977524 cid=zDv*yLKyUA +TRC 2025-03-20 03:32:28.949+01:00 Decoded manifest topics="codex node" tid=6977524 cid=zDv*yLKyUA +TRC 2025-03-20 03:32:28.949+01:00 Retrieving pieces from torrent topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:28.950+01:00 Creating store stream for torrent manifest topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:28.950+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:28.973+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:28.974+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=0 +TRC 2025-03-20 03:32:28.974+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=0 +TRC 2025-03-20 03:32:28.982+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.007+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.008+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=1 +TRC 2025-03-20 03:32:29.008+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=1 +TRC 2025-03-20 03:32:29.014+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.026+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.026+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=2 +TRC 2025-03-20 03:32:29.026+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=2 +TRC 2025-03-20 03:32:29.033+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.056+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.056+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=3 +TRC 2025-03-20 03:32:29.056+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=3 +TRC 2025-03-20 03:32:29.063+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.074+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.074+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=4 +TRC 2025-03-20 03:32:29.074+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=4 +TRC 2025-03-20 03:32:29.081+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.091+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.092+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=5 +TRC 2025-03-20 03:32:29.092+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=5 +TRC 2025-03-20 03:32:29.097+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.119+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.119+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=6 +TRC 2025-03-20 03:32:29.119+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=6 +TRC 2025-03-20 03:32:29.126+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.138+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.139+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=7 +TRC 2025-03-20 03:32:29.139+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=7 +TRC 2025-03-20 03:32:29.145+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.169+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.169+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=8 +TRC 2025-03-20 03:32:29.169+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=8 +TRC 2025-03-20 03:32:29.176+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.195+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.196+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=9 +TRC 2025-03-20 03:32:29.196+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=9 +TRC 2025-03-20 03:32:29.201+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.222+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.223+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=10 +TRC 2025-03-20 03:32:29.223+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=10 +TRC 2025-03-20 03:32:29.230+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.241+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.241+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=11 +TRC 2025-03-20 03:32:29.241+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=11 +TRC 2025-03-20 03:32:29.247+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.270+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.270+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=12 +TRC 2025-03-20 03:32:29.270+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=12 +TRC 2025-03-20 03:32:29.277+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.288+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.288+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=13 +TRC 2025-03-20 03:32:29.288+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=13 +TRC 2025-03-20 03:32:29.295+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.318+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.318+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=14 +TRC 2025-03-20 03:32:29.318+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=14 +TRC 2025-03-20 03:32:29.324+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.336+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.336+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=15 +TRC 2025-03-20 03:32:29.336+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=15 +TRC 2025-03-20 03:32:29.343+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.371+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.371+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=16 +TRC 2025-03-20 03:32:29.371+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=16 +TRC 2025-03-20 03:32:29.378+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.388+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.389+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=17 +TRC 2025-03-20 03:32:29.389+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=17 +TRC 2025-03-20 03:32:29.395+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.418+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.418+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=18 +TRC 2025-03-20 03:32:29.418+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=18 +TRC 2025-03-20 03:32:29.425+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.436+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.436+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=19 +TRC 2025-03-20 03:32:29.436+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=19 +TRC 2025-03-20 03:32:29.443+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.465+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.465+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=20 +TRC 2025-03-20 03:32:29.465+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=20 +TRC 2025-03-20 03:32:29.472+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.484+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.484+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=21 +TRC 2025-03-20 03:32:29.484+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=21 +TRC 2025-03-20 03:32:29.491+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.513+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.514+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=22 +TRC 2025-03-20 03:32:29.514+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=22 +TRC 2025-03-20 03:32:29.518+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.535+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.536+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=23 +TRC 2025-03-20 03:32:29.536+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=23 +TRC 2025-03-20 03:32:29.542+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.564+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.564+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=24 +TRC 2025-03-20 03:32:29.564+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=24 +TRC 2025-03-20 03:32:29.569+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.595+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.595+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=25 +TRC 2025-03-20 03:32:29.595+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=25 +TRC 2025-03-20 03:32:29.602+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.633+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.633+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=26 +TRC 2025-03-20 03:32:29.633+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=26 +TRC 2025-03-20 03:32:29.640+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.663+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.663+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=27 +TRC 2025-03-20 03:32:29.663+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=27 +TRC 2025-03-20 03:32:29.668+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.688+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.688+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=28 +TRC 2025-03-20 03:32:29.688+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=28 +TRC 2025-03-20 03:32:29.695+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.716+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.717+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=29 +TRC 2025-03-20 03:32:29.717+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=29 +TRC 2025-03-20 03:32:29.722+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.737+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.738+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=30 +TRC 2025-03-20 03:32:29.738+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=30 +TRC 2025-03-20 03:32:29.745+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.767+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.768+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=31 +TRC 2025-03-20 03:32:29.768+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=31 +TRC 2025-03-20 03:32:29.773+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.789+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.790+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=32 +TRC 2025-03-20 03:32:29.790+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=32 +TRC 2025-03-20 03:32:29.796+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.818+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.818+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=33 +TRC 2025-03-20 03:32:29.818+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=33 +TRC 2025-03-20 03:32:29.823+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.838+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.838+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=34 +TRC 2025-03-20 03:32:29.838+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=34 +TRC 2025-03-20 03:32:29.845+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.866+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.867+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=35 +TRC 2025-03-20 03:32:29.867+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=35 +TRC 2025-03-20 03:32:29.872+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.890+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.891+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=36 +TRC 2025-03-20 03:32:29.891+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=36 +TRC 2025-03-20 03:32:29.897+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.919+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.920+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=37 +TRC 2025-03-20 03:32:29.920+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=37 +TRC 2025-03-20 03:32:29.926+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.943+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.943+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=38 +TRC 2025-03-20 03:32:29.943+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=38 +TRC 2025-03-20 03:32:29.950+01:00 Waiting for piece... topics="codex restapi" tid=6977524 +TRC 2025-03-20 03:32:29.973+01:00 Fetched complete torrent piece - verifying... topics="codex node" tid=6977524 +TRC 2025-03-20 03:32:29.973+01:00 Piece successfully verified topics="codex node" tid=6977524 pieceIndex=39 +TRC 2025-03-20 03:32:29.973+01:00 Got piece topics="codex restapi" tid=6977524 pieceIndex=39 +INF 2025-03-20 03:32:29.978+01:00 Sent bytes for torrent topics="codex restapi" tid=6977524 infoHash=sha1/04010165844CF71179B42AC3C0462E31EB9E74B1 bytes=10485760 +``` \ No newline at end of file diff --git a/90 Extras/92 Assets/BitTorrent-Upload-Download.svg b/90 Extras/92 Assets/BitTorrent-Upload-Download.svg new file mode 100644 index 0000000..39f149c --- /dev/null +++ b/90 Extras/92 Assets/BitTorrent-Upload-Download.svg @@ -0,0 +1,3 @@ +FILE— OR —InputDictionaryOutputmagnet linkInputDictionaryOutputmagnet link.torrent fileFILE— OR —info hashuploaddownload \ No newline at end of file diff --git a/90 Extras/92 Assets/Codex BitTorrent Streaming.svg b/90 Extras/92 Assets/Codex BitTorrent Streaming.svg new file mode 100644 index 0000000..3b2f83b --- /dev/null +++ b/90 Extras/92 Assets/Codex BitTorrent Streaming.svg @@ -0,0 +1,2 @@ +NetworkStorelocalStore: RepoStoreengine: BlockExcEngineprefetchgetBlockblockpiece aggregationblock-1block-Npiece validationstreamingwaitForNextPieceget piece blockspiece readygetBlockblockREST API/api/codex/v1/torrent/{torrent_info_hash}/network/streamtorrentManifest, codexManifest, pieceValidator \ No newline at end of file