status-im
/
nim-dagger
mirror of https://github.com/status-im/nim-dagger.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
nim-dagger/codex/manifest/types.nim

46 lines
1.5 KiB
Nim
Raw Normal View History

Change every dagger to codex (#102)
2022-05-19 19:56:03 +00:00
## Nim-Codex
Integrate erasure (#73) * wip: adding request for storage endpoint * wire in erasure coding * fix tests for erasure coding * put type definitions into separate file * integrate erasure coding * change run/shutdown to start/stop * temporary sleep, otherwise the fsstore blocks
2022-04-06 00:34:29 +00:00
## Copyright (c) 2022 Status Research & Development GmbH
## Licensed under either of
## * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
## * MIT license ([LICENSE-MIT](LICENSE-MIT))
## at your option.
## This file may not be copied, modified, or distributed except according to
## those terms.
Download files without padding (#218) The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations. Note that the wire format of Manifest was changed, so we need to recreate all BlockStores. * Download without padding * Fixed chunker tests * Chunker: get rid of RabinChunker * Verify offset in the chunker tests * Use manifest.originalBytesPadded in StoreStream.size * StoreStream: replace emptyBlock with zeroMem * Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return * Manifest: verify originalBytes and originalLen on new/encode/decode Also set originalBytes in each Manifest creation/update scenario * Manifest: comments, split code into sections * Reordered parameters to deal with int64 size in 32-bit builds * TestNode.nim: combine Store and Retrieve tests 1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data 2. New test compares only file contents, manifest contents considered an implementation detail 3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch * TestNode.nim: code refactoring * Manifest.add: one more test * Manifest.verify: return Result instead of raising Defect * Node.store: added blockSize parameter
2022-08-24 12:15:59 +00:00
# This module defines Manifest and all related types
Integrate erasure (#73) * wip: adding request for storage endpoint * wire in erasure coding * fix tests for erasure coding * put type definitions into separate file * integrate erasure coding * change run/shutdown to start/stop * temporary sleep, otherwise the fsstore blocks
2022-04-06 00:34:29 +00:00
import std/tables
import pkg/libp2p
import pkg/questionable
const
Repo limits (#319) * initial implementation of repo store * allow isManifest on multicodec * rework with new blockstore * add raw codec * rework listBlocks * remove fsstore * reworking with repostore * bump datastore * fix listBlocks iterator * adding store's common tests * run common store tests * remove fsstore backend tests * bump datastore * add `listBlocks` tests * listBlocks filter based on block type * disabling tests in need of rewriting * allow passing block type * move BlockNotFoundError definition * fix tests * increase default advertise loop sleep to 10 mins * use `self` * add cache quota functionality * pass meta store and start repo * add `CacheQuotaNamespace` * pass meta store * bump datastore to latest master * don't use os `/` as key separator * Added quota limits support * tests for quota limits * add block expiration key * remove unnesesary space * use idleAsync in listBlocks * proper test name * re-add contrlC try/except * add storage quota and block ttl config options * clarify comments * change expires key format * check for block presence before storing * bump datastore * use dht with fixed datastore `has` * bump datastore to latest master * bump dht to latest master
2022-12-03 00:00:55 +00:00
BlockCodec* = multiCodec("raw")
Integrate erasure (#73) * wip: adding request for storage endpoint * wire in erasure coding * fix tests for erasure coding * put type definitions into separate file * integrate erasure coding * change run/shutdown to start/stop * temporary sleep, otherwise the fsstore blocks
2022-04-06 00:34:29 +00:00
DagPBCodec* = multiCodec("dag-pb")
type
ManifestCoderType*[codec: static MultiCodec] = object
DagPBCoder* = ManifestCoderType[multiCodec("dag-pb")]
const
ManifestContainers* = {
$DagPBCodec: DagPBCoder()
}.toTable
type
Manifest* = ref object of RootObj
Download files without padding (#218) The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations. Note that the wire format of Manifest was changed, so we need to recreate all BlockStores. * Download without padding * Fixed chunker tests * Chunker: get rid of RabinChunker * Verify offset in the chunker tests * Use manifest.originalBytesPadded in StoreStream.size * StoreStream: replace emptyBlock with zeroMem * Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return * Manifest: verify originalBytes and originalLen on new/encode/decode Also set originalBytes in each Manifest creation/update scenario * Manifest: comments, split code into sections * Reordered parameters to deal with int64 size in 32-bit builds * TestNode.nim: combine Store and Retrieve tests 1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data 2. New test compares only file contents, manifest contents considered an implementation detail 3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch * TestNode.nim: code refactoring * Manifest.add: one more test * Manifest.verify: return Result instead of raising Defect * Node.store: added blockSize parameter
2022-08-24 12:15:59 +00:00
rootHash*: ?Cid # Root (tree) hash of the contained data set
originalBytes*: int # Exact size of the original (uploaded) file
blockSize*: int # Size of each contained block (might not be needed if blocks are len-prefixed)
blocks*: seq[Cid] # Block Cid
Integrate erasure (#73) * wip: adding request for storage endpoint * wire in erasure coding * fix tests for erasure coding * put type definitions into separate file * integrate erasure coding * change run/shutdown to start/stop * temporary sleep, otherwise the fsstore blocks
2022-04-06 00:34:29 +00:00
version*: CidVersion # Cid version
hcodec*: MultiCodec # Multihash codec
codec*: MultiCodec # Data set codec
case protected*: bool # Protected datasets have erasure coded info
of true:
Enable stylecheck (#353) * applying styleCheck * stuck on vendor folder * Applies style check * Turns styleCheck back off * switches to stylecheck:usages * Fixes empty template casing * rolls up nim-blscurve, nim-datastore, nim-ethers, nim-leopard, and nim-taskpools. * bumps nim-confutils and removes unused import from fileutils.nim * Unused using in fileutils.nim is required by CI * Reverts bump of nim-confutils module
2023-03-10 07:02:54 +00:00
ecK*: int # Number of blocks to encode
ecM*: int # Number of resulting parity blocks
Integrate erasure (#73) * wip: adding request for storage endpoint * wire in erasure coding * fix tests for erasure coding * put type definitions into separate file * integrate erasure coding * change run/shutdown to start/stop * temporary sleep, otherwise the fsstore blocks
2022-04-06 00:34:29 +00:00
originalCid*: Cid # The original Cid of the dataset being erasure coded
originalLen*: int # The length of the original manifest
else:
discard