From befb3beb2701d0fedab39823160d19577885d073 Mon Sep 17 00:00:00 2001 From: Marcin Czenko Date: Wed, 28 May 2025 16:36:59 +0200 Subject: [PATCH] Adds docs related to using magnet links and torrent files to download content --- 10 Notes/B-encoding using Python.md | Bin 0 -> 1710 bytes 10 Notes/Magnet Links.md | 2 +- .../PR 1179 - checked exceptions in stores.md | 36 ++++++++++++ ...ad BitTorrent content with Codex - demo.md | 52 ++++++++++++++---- 10 Notes/Using Python on macOS.md | 52 ++++++++++++++++++ ...Download Content from the Codex Network.md | Bin 0 -> 5723 bytes 6 files changed, 131 insertions(+), 11 deletions(-) create mode 100644 10 Notes/B-encoding using Python.md create mode 100644 10 Notes/PR 1179 - checked exceptions in stores.md create mode 100644 10 Notes/Using Python on macOS.md create mode 100644 10 Notes/Using Torrent Files to Download Content from the Codex Network.md diff --git a/10 Notes/B-encoding using Python.md b/10 Notes/B-encoding using Python.md new file mode 100644 index 0000000000000000000000000000000000000000..2d0302726aaf0ea94c249a84ed68d8273e829978 GIT binary patch literal 1710 zcmb7F&yUdwX)=ciPu$z-M~2(?m+sqgtkWumv$JotkQeY`Z(1b*%Z^pg zU!I>Y9OpHMOVM>OYTOO)_^<~pc74a+@7C2IdILtmJJyMs84g@lsz!%;0dLd@6_d~i zS%Yag9JaT7;3>4m^g2B~y%5?gE`({va#5+>>5dJRkkwCIzE!6zj!M=9URai1Q`53; znkewAn$aqZ14bO@`8a@({b=k_OFlA$vkDn`CpltM0y8U`-S;(M*+7LV2O z)N!`ktrR=wX9vI)vQcgdQy)himlvkRG4W^|MByVlE!mFaq(f=32N|BbAVt1RBHzm+KU}%lVCcI?2b$ZLV(ry?Q{T@zECIn8D?DlY--W#|%mafRRk!xrMwX;Cpy8jt^ zL|S+I6kZJ@w?Kcu=m=W(7GT|5VYMsQpt^U2)4d~*?j0d??*iUA!Zu~UMQos1E(1!v z1Sv*>DDqg{(1^#3hJF~w2@jGwA$83UK;zxs9$~iy)f<6QBBVCMSQ&kM{ZTDufZtPN zUYjCAEvm_kq0TUrj9P(I2Ax#hSaVP)ApWlc`Vm$v_urIXfBfPzJNWcTaPP}Mf3pMICc`H~#}5cT!!+wT+q23KoYk!C z_#-HJ#U}Zm|49Rq1E;tIOcL%i9EDv$%YOs}dQ$yq$e`&ECN6Qc+~b?ThJoSbMkMD{ f^Z7Zzcv$q<_extgsNMMXv_hg%91ehSYQFdvI?gC& literal 0 HcmV?d00001 diff --git a/10 Notes/Magnet Links.md b/10 Notes/Magnet Links.md index 24c6154..3556af1 100644 --- a/10 Notes/Magnet Links.md +++ b/10 Notes/Magnet Links.md @@ -26,7 +26,7 @@ magnet:?xt=urn:btmh:122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc5164 **Hybrid** ``` -magnet:?xt=urn:btmh:122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1&urn:btih:1902d602db8c350f4f6d809ed01eff32f030da95 +magnet:?xt=urn:btmh:122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1&xt=urn:btih:1902d602db8c350f4f6d809ed01eff32f030da95 ``` `122003ee5d27d20f46c2395fb00e407a85f87c885e2e072d01f4e4bc516474203fe1` is a SHA-256 multihash: diff --git a/10 Notes/PR 1179 - checked exceptions in stores.md b/10 Notes/PR 1179 - checked exceptions in stores.md new file mode 100644 index 0000000..e649aa4 --- /dev/null +++ b/10 Notes/PR 1179 - checked exceptions in stores.md @@ -0,0 +1,36 @@ +--- +tags: + - codex/pulls +related-to: + - "[[Codex PRs]]" +--- +https://github.com/codex-storage/nim-codex/pull/1179 + +Two things to be handled in the follow up PR: + +- better signalization in the test `Should propagate cancellation error immediately` in `tests/codex/utils/testsafeasynciter.nim`. See the review comment https://github.com/codex-storage/nim-codex/pull/1179#discussion_r2100831743 +- check what to do with `Error: Exception can raise an unlisted exception: Exception` - see discussion in https://github.com/codex-storage/nim-codex/pull/1179#discussion_r2100873891 + +For reference, here is the essence of the `Error: Exception can raise an unlisted exception: Exception` thing: + +```nim +import pkg/chronos +# the problem does not occur when using std/asyncdispatch +# import std/asyncdispatch + +# without "raises: []" it fails with "Error: Exception can raise an unlisted exception: Exception" +#func getIter(): (iterator (): int {.gcsafe.}) = +func getIter(): (iterator (): int {.gcsafe, raises: [].}) = + return iterator (): int = + yield 1 + yield 2 + yield 3 + +proc f1() = + let iter = getIter() + + proc genNext(): Future[int] {.async.} = + iter() +``` + +See also https://github.com/nim-lang/Nim/issues/3772. \ No newline at end of file diff --git a/10 Notes/Upload and download BitTorrent content with Codex - demo.md b/10 Notes/Upload and download BitTorrent content with Codex - demo.md index 85a5239..af57551 100644 --- a/10 Notes/Upload and download BitTorrent content with Codex - demo.md +++ b/10 Notes/Upload and download BitTorrent content with Codex - demo.md @@ -11,10 +11,10 @@ task codex, "build codex binary": In the example session, we will use two nodes: -- node-1 will be where we will upload (seed) the content -- node-2 will be from where we will be downloading the previously uploaded content. +- `node-1` will be where we will upload (seed) the content +- `node-2` will be from where we will be downloading the previously uploaded content. -Start node-1: +Start `node-1`: ```bash ./build/codex --data-dir=./data-1 --listen-addrs=/ip4/127.0.0.1/tcp/8081 --api-port=8001 --nat=none --disc-port=8091 --log-level=TRACE @@ -27,7 +27,7 @@ Generate fresh content: dd if=/dev/urandom of=./data10M.bin bs=10M count=1 ``` -Upload content to node-1: +Upload content to `node-1`: ```bash curl -X POST \ @@ -45,15 +45,21 @@ Set `info_hash` env var: export INFO_HASH=11144249FFB943675890CF09342629CD3782D107B709 ``` -Start node-2: +Or if you want to use a magnet file, create a (text) file (e.g. `magnet-v1-test.txt`) and make sure you include your INFO hash it. The response returned after uploading the content is a multihash. To get a regular hash, you just have to skip the first 4 characters: + +``` +magnet:?xt=urn:btih:4249FFB943675890CF09342629CD3782D107B709 +``` + +Start `node-2`: ```bash ./build/codex --data-dir=./data-2 --listen-addrs=/ip4/127.0.0.1/tcp/8082 --api-port=8002 --nat=none --disc-port=8092 --log-level=TRACE ``` -Make sure that node-1 and node-2 are connect (should be automatic but sometimes we need to trigger it when running on localhost without nat). +Now, let's make sure that `node-1` and `node-2` are connected. -Get the peerId from node-1 +Get the peerId from `node-1` ```bash curl -H 'Accept: text/plain' 'http://localhost:8001/api/codex/v1/peerid' --write-out '\n' @@ -66,7 +72,7 @@ Here it is also good to use env var: export PEERID_NODE1=$(curl -H 'Accept: text/plain' 'http://localhost:8001/api/codex/v1/peerid') ``` -Connect node-2 to node-1: +Connect `node-2` to `node-1`: ```bash curl "http://localhost:8002/api/codex/v1/connect/${PEERID_NODE1}?addrs=/ip4/127.0.0.1/tcp/8081" @@ -79,12 +85,35 @@ Make sure `INFO_HASH` env var is set: export INFO_HASH=11144249FFB943675890CF09342629CD3782D107B709 ``` -Stream the content from node-2: +Stream the content from `node-2` using `INFO_HASH` directly use: ```bash curl "http://localhost:8002/api/codex/v1/torrent/${INFO_HASH}/network/stream" -o "${INFO_HASH}.bin" ``` +and to stream the content from `node-2` using the magnet link, use: + +```bash +curl -X POST \ + http://localhost:8001/api/codex/v1/torrent/magnet \ + -H 'Content-Type: text/plain' \ + -w '\n' \ + -T magnet-v1-test.txt \ + -o magnet-v1-content.bin +``` +(you can change the name under which the downloaded content will be saved, here it is `magnet-v1-content.bin`). + +To use magnet links, you do not have to create a file. You can just directly paste the magnet link text as follows: + +```bash +curl -X POST \ + http://localhost:8001/api/codex/v1/torrent/magnet \ + -H 'Content-Type: text/plain' \ + -w '\n' \ + -d 'magnet:?xt=urn:btih:4249FFB943675890CF09342629CD3782D107B709' \ + -o magnet-v1-content.bin +``` + And to get just the torrent manifest: ```bash @@ -147,7 +176,10 @@ curl "http://localhost:8002/api/codex/v1/torrent/${INFO_HASH}/network/manifest" } ``` -And here is the streaming log (node-2) for the reference: +You can also use a torrent file when downloading the content from the Codex network. Please check [[Using Torrent Files to Download Content from the Codex Network]]. +### Appendix + +The streaming log (node-2) for the reference: ```bash TRC 2025-03-20 03:32:28.939+01:00 torrent requested: topics="codex restapi" tid=6977524 multihash=sha1/04010165844CF71179B42AC3C0462E31EB9E74B1 diff --git a/10 Notes/Using Python on macOS.md b/10 Notes/Using Python on macOS.md new file mode 100644 index 0000000..03fc6b1 --- /dev/null +++ b/10 Notes/Using Python on macOS.md @@ -0,0 +1,52 @@ +There are some [[Modern Python Tools]], but I tend to rely on the good old [pyenv](https://github.com/pyenv/pyenv) and native python environments. + +### Python Version + +I use `pyenv` to switch between python versions. +I installed it with: + +```bash +brew install pyenv +``` + +Using `pyenv` is convenient. + +You can check existing versions with: + +```bash +# checks the most recent available version starting with 3 +pyenv latest -k 3 + +# checks the most recent available version starting with 3.12 +pyenv latest -k 3.12 +``` + +Now to install the chosen python version, just run: + +```bash +pyenv install 3.13.1 +``` + +When you run a Python command, `pyenv` will look for a `.python-version` file in the current directory and each parent directory. If no such file is found in the tree, `pyenv` will use the global Python version specified with `pyenv global`. A version specified with the `PYENV_VERSION` environment variable takes precedence over local and global versions. + +Run `pyenv versions` for a list of available Python versions. To read more, run `pyenv local --help`. + +Running `pyenv local 3.13.1` will create the `.python-version` file in the current directory. + +### Python Environment + +On my Mac I keep python environments in `~/python-venvs`. Now to create a new environment, run: + +```bash +python -m venv ~/python-venvs/ +``` + +To activate and deactivate given python environment, run: + +```bash +# to activate +source ~/python-venvs//bin/activate + +# to deactivate +deactivate +``` diff --git a/10 Notes/Using Torrent Files to Download Content from the Codex Network.md b/10 Notes/Using Torrent Files to Download Content from the Codex Network.md new file mode 100644 index 0000000000000000000000000000000000000000..e398b0d166a819257ec1c48da3f344e4028bd972 GIT binary patch literal 5723 zcmd^D%Z}T~70qgZQL@b@0S0AZ0M7s|iIhmafDVRwP21zKJ8mV8-P4FlWKrTYMapEc z<@ThvL4GL9AiHd{%Wvcta&8qVQF8Z8lmtkC0DACIRrl7tr_Mc9lQ>~=DVQ5aHzEo| z-LR7Z@sA(?G(5l@EKKs7EPZiG3%?!MPy!RKG;{G`t=Fvr!h} zNOc|ra+>>}yTf35Bf^mW^F{~mKG9u8@)GS+7)n!7d*~u@qOhn$`1|&QwrS#uLHjHDmyBgHgH6QnupmGKfSi z5!~bRP}JE32@^;aTg7OGMmljh+@Ufl91JY!_Hq^OxZUT z#+Fj9`;A5zyF6URsqA+R!))+%(4a`ZYuuQPVh+^2cE%nLH9l(7+qLL3zFvoc%ViKp zjo6h!)>4TutRAc3%k4)&iYW%vvag(t0L#;=$0awvuFr$0n)lhT$KOQx^_1=2S4`8i zCasojSw^qdwN1OzF`UuF>WsUN)o!-C-QL*lje6#2G^Qz6D$nB$MGUz~1R#do>+lkw z;Z39$U=D92Jn#i#u#1;?r@-*!`Q*iK&zy_1)1zEa(XQSX;^w^}ZhZRP*3w-gUu})T ztt$H+`}S?+JEWmX%jOITt@c^<7?-Li)H~1U*ZF)g2o0JKyS>AGD58ZVBc@?>?M|C| ztA^`9xZ+vyw%Kl)mbL#fMX$@eIYuHUo91LZa9T!pII_)g&lyg-UCZp4y_VA(J8ffP zOuAOvYAYvmL)5P|vYc+SgN(OEPIKh6Y}>IrR>v}$W@ltsZQE#%yW_!OKhqhtyX}G1 z9rWzBF)-}`?6oE%!x(qQLvvvF+D&U<^bY&khG~yGBgYt8Mhjy%tzmm$JMeyFjs~q> zr)dm2&GxXGPw#3wb3!!cUB)y+;Ib6J+zIro~=hM*}_w9SKd{?>C7@vYJa~@rDFjr0?0DP82 zir;nhDR9KVCp;ulyajjM@GzhnP^wS`yD2tBh0^Qc+cb`L;DhHmX70yG^`21VL3kKD zRrN|E=i}o+Q*=23FGZIlpi*=>4kcG71ZYazNs+g_NUB0ogj%svoOr0lc^bE+k&1vw z87d$On}i(bE2eit=nkn1Y)XZTs%NPacU_E!EQYsKw)EJj5>$Yw;A;JX?4uZ=Wr-GZ zOsVS2iV}xDs*CbE5o@%Ec}4$)#|Kz~t=W#li@IGWIL%Nw2g2o~lHv#s<|e(sr7FP_ zxSVc9$_E~A3B2U#36u!}rNXr2ME5bK?S`30+5!a?^ChLD;a-Zt)Rq?cB z9kv|OCCGc=Qo6WKz>F4HsU0Ih2o)jV1NUwMVBwazOAen&I@U;Z(87nnSu9ZGV46mt z_@opkuJr@t$S(R2Z%B8F_(8U!2{e;tl|$Kj6L997Ke;dX@jMlYD!XL8(0O(CT0qRm zW%(gnWl%Mcr8%mU7z`kHz)_jlzP%P}5|NVR6XuLLMhkKX6I3#j`#ynr3+($~qK8Cm zUqCFua}Z(}ZwPA?5W*vM|K~xzTUXzsGjd^Fk{ADm94lm5P079=)@HQcen4OCtxZ;o zSy_(MC>W&DbkQNqV-Mjjg?qgNr7t9&cp6H@J_0EYJdgxbVo1>~vB9hqi+C-fG!0q0 zw0q)QEO|ggz zGA#4~Rxyzb7JNZa5BgKu8{U|(r&(1Y5?R;&=bK5ruOZ!9BA7Y@j}QFKPzQ&`%pw>9sK{LWhxb@ znUc1^d5DrCK!MswBQbgpe)4^!`a$_u1%-Hq5Y5^7ty}{3shb2S0h?vu;$#751xxJm zplvQM2l@nnke#7yE%F{$S9L~2ua5^3T_yyen5c?MCjme%YQ{s#3#PD{FGLzZGzGTw zaRyo09z0-&Us?hM0jjDr7bULqtUj#-uie+!4f6e!u=)gj)9CLA&usR~Fb_1yL|Z>i zyGH-f-~aq4d<^~f3I6{22j{K6!|zOAO(v&LadnIDseWR=dG+NNkM2g|BYpp~zx`fc z;FmgG;yc9mNBH9N^VjpQK0g1czWK=)_D7%o<&XMC|Mrp}C!r9#P03!k)!~X(4>>U) zQV*BrQn37k9Y6Pct0THR-|-8(=k-j_%MCw)U|Ggl=#feWK8OW!Txl5rz$~DlxjIqP zhi3&7?&$G6ld^ca%kDnGMdeCX)Xv!+WI@7O078_JBmb-7;GTn6di^_jP7pO@dEeWXCv1;HS DL^3l5 literal 0 HcmV?d00001