logos-storage/logos-storage-docs-obsidian
1
0
Fork 0
mirror of https://github.com/logos-storage/logos-storage-docs-obsidian.git synced 2026-01-03 05:43:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

Updates Codex Encryption docs

Browse Source
This commit is contained in:
Marcin Czenko 2025-07-08 03:14:42 +02:00
parent 453906fd47
commit 7a2be687f2
No known key found for this signature in database
GPG Key ID: 33DEA0C8E30937C0
3 changed files with 120 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

83
10 Notes/Codex Encryption Design.md
View File

@ -29,43 +29,10 @@ For some introduction and examples on BearSSL, please consult:
- [[How to create a hash using BearSSL]]
- [[How to encrypt and decrypt content using symmetric encryption in BearSSL]]
Before document design considerations for the content encryption in the Codex client, let's first see how to use BearSSL primitives to encrypt and decrypt some content:
In Codex, we use a slightly modified AES-CBC mode of operation:
```nim
import std/sequtils
import bearssl/blockx
import stew/byteutils
import ./rng
var plaintext = "0123456789abcdef".toBytes
echo "plaintext: ", plaintext.toHex
let key = newSeqWith(16, Rng.instance.rand(uint8.high).byte)
let ive = newSeqWith(16, Rng.instance.rand(uint8.high).byte)
let ivd = ive
echo "ive: ", ive.toHex
var encCtx: AesBigCbcencKeys
aesBigCbcencInit(encCtx, addr key[0], 16.uint)
aesBigCbcencRun(encCtx, addr ive[0], addr plaintext[0], 16.uint)
echo "Encrypted: ", plaintext.toHex
echo "ivd: ", ivd.toHex
var decCtx: AesBigCbcdecKeys
aesBigCbcdecInit(decCtx, addr key[0], 16.uint)
aesBigCbcdecRun(decCtx, addr ivd[0], addr plaintext[0], 16.uint)
echo "Decrypted: ", plaintext.toHex
```
Important to notice here is that `aesBigCbcencRun` will mutate the provided initialization vector `IV` so that it is ready to use for the subsequent chunk of data - a classical CBC mode for AES. Yet, for Codex, we use slightly modified scheme as already shown above.
For codex:
1. we first generate a `MASTER_KEY` - which will be returned to the user
2. from the `MASTER_KEY`, for each block, we derive the corresponding block level encryption key `blockKEY` and block level initialization vector `blockIV` as shown in the proposal above
1. we first generate a random `MASTER_KEY` - this key will be returned to the user,
2. from the `MASTER_KEY`, for each block, we derive the corresponding block level encryption key `blockKEY` and block level initialization vector `blockIV` as shown in the proposal above,
3. using the derived `blockKEY` and `blockIV`, we then encrypt the block using the BearSSL encryption primitives as demonstrated above.
As we see above, the block index is used in the process of the key and initialization vector derivation. For this reason we also need to remember to convert the block index to a byte representation - we use big-endian ordering. For this conversion a very simple function can be used:
@ -97,12 +64,15 @@ let masterKey = newSeqWith(32, Rng.instance.rand(uint8.high).byte)
let blockIndex = 1.uint32
let blockIndexArray = toBytes(blockIndex, bigEndian)
const KEY_SIZE = 24 # 192 bits for AES-192
const IV_SIZE = 16 # 128 bits
const DefaultBlockSize* = uint 1024 * 64 # as used in Codex
const
KEY_SIZE = 24 # 192 bits for AES-192
IV_SIZE = 16 # 128 bits
KeyDerivationIdentifier = "aes192_block_key".toBytes
IvDerivationIdentifier = "aes192_block_iv".toBytes
DefaultBlockSize* = uint 1024 * 64 # as used in Codex
let keyForBlock = hash(addr sha256Vtable, masterKey & @[byte(0x01)] & blockIndexArray.toSeq)[0 ..< KEY_SIZE]
let ivForBlock = hash(addr sha256Vtable, masterKey & @[byte(0x02)] & blockIndexArray.toSeq)[0 ..< IV_SIZE]
let keyForBlock = hash(addr sha256Vtable, masterKey & KeyDerivationIdentifier & blockIndexArray.toSeq)[0 ..< KEY_SIZE]
let ivForBlock = hash(addr sha256Vtable, masterKey & IvDerivationIdentifier & blockIndexArray.toSeq)[0 ..< IV_SIZE]
var plaintext = newSeqWith(DefaultBlockSize.int, Rng.instance.rand(uint8.high).byte)
@ -113,7 +83,7 @@ let encBuffer = plaintext
# encryption
var encCtx: AesBigCbcencKeys
aesBigCbcencInit(encCtx, addr key[0], key.len.uint)
aesBigCbcencRun(encCtx, addr ive[0], addr encBuffer[0], ive.len.uint)
aesBigCbcencRun(encCtx, addr ive[0], addr encBuffer[0], encBuffer.len.uint)
assert encBuffer != plaintext, "Encryption failed, output should differ from input!"
@ -122,17 +92,40 @@ let ivd = ivForBlock
var decCtx: AesBigCbcdecKeys
aesBigCbcdecInit(decCtx, addr key[0], key.len.uint)
aesBigCbcdecRun(decCtx, addr ivd[0], addr encBuffer[0], ivd.len.uint)
aesBigCbcdecRun(decCtx, addr ivd[0], addr encBuffer[0], encBuffer.len.uint)
assert encBuffer == plaintext, "Decryption failed, output should match input!"
```
where `rng` and `hash` are defined as shown in [[How to generate a random number using BearSSL]] and [[How to create a hash using BearSSL]].
In a similar way we will proceed with other block.
In a similar way we will proceed with the following, successive blocks.
More data to come.
### API
All data uploaded to Codex will, be default, be encrypted.
The standard upload API - `/api/codex/v1/data` - after successful upload, returns the content identifier of the uploaded content, followed by the hex encoded master encryption key, separated by a single comma character `:`, eg:
```bash
curl -X POST \
http://localhost:8001/api/codex/v1/data \
-H 'Content-Type: application/octet-stream' \
-H 'Content-Disposition: filename="enc.txt"' \
-w '\n' \
-T enc.txt
zDvZRwzm22eSYNdLBuNHVi7jSTR2a4n48yy4Ur9qws4vHV6madiz:541cc266b2ef426486cd981f2d7429347c68113bda2a80d21c9f18e250bfdaff
```
When downloading, the user is allowed to either retrieve the encrypted content, and then use an external utility (to be provided) to decrypt it, or the user can provide the encryption master key, which then will be used by the Codex client to decrypt the content, which then will be returned to the user:
```bash
export CID="zDvZRwzm22eSYNdLBuNHVi7jSTR2a4n48yy4Ur9qws4vHV6madiz"
export KEY="541cc266b2ef426486cd981f2d7429347c68113bda2a80d21c9f18e250bfdaff"
curl "http://localhost:8001/api/codex/v1/data/${CID}/network/stream?key=${KEY}" -o "dec.txt"
```
Notice that when no encryption key is provided, the returned content size will always be multiply of the block size.
### Links
- [bearssl](https://bearssl.org/)

65
10 Notes/How to check out git tags.md Normal file
View File

@ -0,0 +1,65 @@
---
link: https://graphite.dev/guides/how-to-check-out-git-tags
---
```bash
# fetch tags
git fetch --tags
# list tags
git tag
# search
git tag -l <pattern>
```
## [Basic tag checkout](https://graphite.dev/guides/how-to-check-out-git-tags#basic-tag-checkout)
- **Git checkout tag**: To checkout a tag, use the `git checkout` command followed by the tag name:
`git checkout <tag-name>`
This command switches your working directory to the state of the specified tag.
- **Git checkout a tag as new branch**: If you want to make changes starting from a tag, it's best to checkout the tag into a new branch:
`git checkout -b <new-branch-name> <tag-name>`
### [Working with remote tags](https://graphite.dev/guides/how-to-check-out-git-tags#working-with-remote-tags)
- **Git checkout tag from remote**: To checkout a tag that exists in the remote repository but not locally, first fetch the tags:
`git fetch --tags`
Then, checkout the tag using the `git checkout <tag-name>` command.
- **Git checkout remote tag as branch**: After fetching tags, you can create a new branch from a remote tag similarly to how you would with a local tag:
`git checkout -b <branch-name> <tag-name>`
### [Advanced tag operations](https://graphite.dev/guides/how-to-check-out-git-tags#advanced-tag-operations)
- **Git checkout latest tag**: To checkout the most recent tag, first fetch the tags from remote:
`git fetch --tags`
Then find the latest tag with a combination of the `git describe` and `git rev-list` commands. You can save this tag to a local variable for use in the next step:
`latestTag=$(git describe --tags $(git rev-list --tags --max-count=1))`
Once you have the latest tag, check out as normal using the variable we just created:
`git checkout $latestTag`
- **Git Checkout tagged Version/branch/commit**: These operations are fundamentally similar, as they involve checking out a specific state identified by a tag. Use the `git checkout <tag-name>` command for a version or commit, and if you want to start a new branch from a tagged commit, use `git checkout -b <new-branch-name> <tag-name>`.
### [Handling issues and special cases](https://graphite.dev/guides/how-to-check-out-git-tags#handling-issues-and-special-cases)
- **Git checkout tag without detached HEAD**: When you checkout a tag directly, Git puts your repository in a "detached HEAD" state. To avoid this, checkout the tag into a new branch:
`git checkout -b <new-branch-name> <tag-name>`
For further reading on this state, please see the guide on [resolving a detached HEAD state](https://graphite.dev/guides/how-to-resolve-detached-head-state-in-git).
- **Git checkout tag not working**: If you encounter issues checking out a tag, ensure the tag exists (`git tag`) and that you have fetched the latest tags from the remote repository (`git fetch --tags`).

22
10 Notes/How to encrypt and decrypt content using symmetric encryption in BearSSL.md
View File

@ -8,24 +8,36 @@ import stew/byteutils
import ./rng
const
KeySize = 24 # 192 bits for AES-192
IvSize = 16 # fixed - 128 bits - notice that the length og the content to be
# encrypted/decrypted must be multiply of 16
var plaintext = "0123456789abcdef".toBytes
echo "plaintext: ", plaintext.toHex
let key = newSeqWith(16, Rng.instance.rand(uint8.high).byte)
let ive = newSeqWith(16, Rng.instance.rand(uint8.high).byte)
let key = newSeqWith(KeySize, Rng.instance.rand(uint8.high).byte)
let ive = newSeqWith(IvSize, Rng.instance.rand(uint8.high).byte)
let ivd = ive
echo "ive: ", ive.toHex
var encCtx: AesBigCbcencKeys
aesBigCbcencInit(encCtx, addr key[0], 16.uint)
aesBigCbcencRun(encCtx, addr ive[0], addr plaintext[0], 16.uint)
aesBigCbcencRun(encCtx, addr ive[0], addr plaintext[0], plaintext.len.uint)
echo "Encrypted: ", plaintext.toHex
echo "ivd: ", ivd.toHex
var decCtx: AesBigCbcdecKeys
aesBigCbcdecInit(decCtx, addr key[0], 16.uint)
aesBigCbcdecRun(decCtx, addr ivd[0], addr plaintext[0], 16.uint)
aesBigCbcdecRun(decCtx, addr ivd[0], addr plaintext[0], plaintext.len.uint)
echo "Decrypted: ", plaintext.toHex
```
```
As stated in the comment, the following note must be emphasized:
>[!note]
>The size of the `IV` vector must be $16$ bytes long. The data to be encrypted or decrypted must have length that is multiple of $16$
Also important to notice is that `aesBigCbcencRun` will mutate the provided initialization vector `IV` so that it is ready to be used for the subsequent chunk of data - a classical CBC mode for AES.