codex-storage
/
nim-codex
mirror of https://github.com/codex-storage/nim-codex.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
nim-codex/codex/slots/converters.nim

89 lines
2.5 KiB
Nim
Raw Normal View History

Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
## Nim-Codex
## Copyright (c) 2024 Status Research & Development GmbH
## Licensed under either of
## * Apache License, version 2.0, ([LICENSE-APACHE](LICENSE-APACHE))
## * MIT license ([LICENSE-MIT](LICENSE-MIT))
## at your option.
## This file may not be copied, modified, or distributed except according to
## those terms.
import std/sequtils
import pkg/libp2p
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
import pkg/stew/arrayops
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
import pkg/questionable
import pkg/questionable/results
import pkg/poseidon2
import pkg/poseidon2/io
import ../codextypes
import ../merkletree
import ../errors
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
import ../utils/digest
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
func toCid(hash: Poseidon2Hash, mcodec: MultiCodec, cidCodec: MultiCodec): ?!Cid =
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
let
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
mhash = ? MultiHash.init(mcodec, hash.toBytes).mapFailure
treeCid = ? Cid.init(CIDv1, cidCodec, mhash).mapFailure
success treeCid
proc toPoseidon2Hash(cid: Cid, mcodec: MultiCodec, cidCodec: MultiCodec): ?!Poseidon2Hash =
if cid.cidver != CIDv1:
return failure("Unexpected CID version")
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
if cid.mcodec != cidCodec:
return failure("Cid is not of expected codec. Was: " & $cid.mcodec & " but expected: " & $cidCodec)
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
let
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
mhash = ? cid.mhash.mapFailure
bytes: array[32, byte] = array[32, byte].initCopyFrom(mhash.digestBytes())
hash = ? Poseidon2Hash.fromBytes(bytes).toFailure
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
success hash
func toCellCid*(hash: Poseidon2Hash): ?!Cid =
toCid(hash, Pos2Bn128MrklCodec, CodexSlotCellCodec)
func fromCellCid*(cid: Cid): ?!Poseidon2Hash =
toPoseidon2Hash(cid, Pos2Bn128MrklCodec, CodexSlotCellCodec)
func toSlotCid*(hash: Poseidon2Hash): ?!Cid =
toCid(hash, multiCodec("identity"), SlotRootCodec)
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
func toSlotCids*(slotRoots: openArray[Poseidon2Hash]): ?!seq[Cid] =
success slotRoots.mapIt( ? it.toSlotCid )
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
func fromSlotCid*(cid: Cid): ?!Poseidon2Hash =
toPoseidon2Hash(cid, multiCodec("identity"), SlotRootCodec)
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
func toVerifyCid*(hash: Poseidon2Hash): ?!Cid =
toCid(hash, multiCodec("identity"), SlotProvingRootCodec)
func fromVerifyCid*(cid: Cid): ?!Poseidon2Hash =
toPoseidon2Hash(cid, multiCodec("identity"), SlotProvingRootCodec)
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
func toEncodableProof*(
proof: Poseidon2Proof): ?!CodexProof =
let
encodableProof = CodexProof(
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
mcodec: multiCodec("identity"),
Contracts handler (#672) * get rid of unneeded files * don't reuse batch callback * move out storage contract handlers * wip * add storage handler, to handle storage contracts interactions * split out node tests * a bit more cleanup * remove storage handler, move back into node * add missing raises: * wip: add support for rebuilding slot * split out tests * wip * rework indexing strategy to return an iterator * convert to seq * minor api change (meh, might revert) * steping strategy should not die * allow fetching batches for manifests and indicies * restored expiry update * restored expiry update functionality * avoid closing datastores twice * correct wrong rename * fixes sigsegv
2024-01-15 16:45:04 +00:00
index: proof.index,
nleaves: proof.nleaves,
path: proof.path.mapIt( @( it.toBytes ) ))
success encodableProof
func toVerifiableProof*(
proof: CodexProof): ?!Poseidon2Proof =
let
wire in circom backend (#698) * wire in circom backend * should contain leafs * adding circom compad and circuits deps * update windows build * fix windows build * improve test names * move proving defaults to codextypes * remove unnedded inmports and move defaults to codextypes * capture error code on backend failure
2024-02-09 21:40:30 +00:00
nodes = proof.path.mapIt(
? Poseidon2Hash.fromBytes(it.toArray32).toFailure
)
Poseidon2Proof.init(
index = proof.index,
nleaves = proof.nleaves,
nodes = nodes)