nim-codex/tests/codex/examples.nim

85 lines
2.4 KiB
Nim
Raw Permalink Normal View History

2021-04-07 10:19:00 +00:00
import std/random
import std/sequtils
import pkg/libp2p
2021-04-07 10:19:00 +00:00
import pkg/nitro
2022-03-23 12:57:48 +00:00
import pkg/stint
2022-05-19 19:56:03 +00:00
import pkg/codex/rng
import pkg/codex/stores
import pkg/codex/blocktype as bt
import pkg/codex/sales
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
import pkg/codex/merkletree
import ../examples
export examples
2021-04-07 10:19:00 +00:00
proc example*(_: type EthAddress): EthAddress =
EthPrivateKey.random().toPublicKey.toAddress
2021-04-07 14:53:00 +00:00
proc example*(_: type UInt48): UInt48 =
# workaround for https://github.com/nim-lang/Nim/issues/17670
uint64.rand mod (UInt48.high + 1)
2021-04-08 13:05:04 +00:00
proc example*(_: type Wallet): Wallet =
Wallet.init(EthPrivateKey.random())
2021-04-19 14:37:38 +00:00
proc example*(_: type WalletRef): WalletRef =
WalletRef.new(EthPrivateKey.random())
proc example*(_: type SignedState): SignedState =
2021-04-08 13:05:04 +00:00
var wallet = Wallet.example
let hub, asset, receiver = EthAddress.example
let chainId, amount = UInt256.example
2021-04-07 14:53:00 +00:00
let nonce = UInt48.example
let channel = wallet.openLedgerChannel(hub, chainId, nonce, asset, amount).get
wallet.pay(channel, asset, receiver, amount).get
2021-04-08 07:45:41 +00:00
proc example*(_: type Pricing): Pricing =
Pricing(
address: EthAddress.example,
price: uint32.rand.u256
2021-04-08 07:45:41 +00:00
)
proc example*(_: type bt.Block): bt.Block =
let length = rand(4096)
let bytes = newSeqWith(length, rand(uint8))
bt.Block.new(bytes).tryGet()
proc example*(_: type PeerId): PeerId =
let key = PrivateKey.random(Rng.instance[]).get
PeerId.init(key.getPublicKey().get).get
proc example*(_: type BlockExcPeerCtx): BlockExcPeerCtx =
BlockExcPeerCtx(id: PeerId.example)
2021-04-26 14:34:04 +00:00
proc example*(_: type Cid): Cid =
bt.Block.example.cid
2022-03-30 10:51:28 +00:00
proc example*(_: type MultiHash, mcodec = Sha256HashCodec): MultiHash =
let bytes = newSeqWith(256, rand(uint8))
MultiHash.digest($mcodec, bytes).tryGet()
2022-03-30 10:51:28 +00:00
proc example*(_: type Availability): Availability =
Availability.init(
totalSize = uint16.example.u256,
freeSize = uint16.example.u256,
duration = uint16.example.u256,
minPrice = uint64.example.u256,
maxCollateral = uint16.example.u256
)
[marketplace] Availability improvements (#535) ## Problem When Availabilities are created, the amount of bytes in the Availability are reserved in the repo, so those bytes on disk cannot be written to otherwise. When a request for storage is received by a node, if a previously created Availability is matched, an attempt will be made to fill a slot in the request (more accurately, the request's slots are added to the SlotQueue, and eventually those slots will be processed). During download, bytes that were reserved for the Availability were released (as they were written to disk). To prevent more bytes from being released than were reserved in the Availability, the Availability was marked as used during the download, so that no other requests would match the Availability, and therefore no new downloads (and byte releases) would begin. The unfortunate downside to this, is that the number of Availabilities a node has determines the download concurrency capacity. If, for example, a node creates a single Availability that covers all available disk space the operator is willing to use, that single Availability would mean that only one download could occur at a time, meaning the node could potentially miss out on storage opportunities. ## Solution To alleviate the concurrency issue, each time a slot is processed, a Reservation is created, which takes size (aka reserved bytes) away from the Availability and stores them in the Reservation object. This can be done as many times as needed as long as there are enough bytes remaining in the Availability. Therefore, concurrent downloads are no longer limited by the number of Availabilities. Instead, they would more likely be limited to the SlotQueue's `maxWorkers`. From a database design perspective, an Availability has zero or more Reservations. Reservations are persisted in the RepoStore's metadata, along with Availabilities. The metadata store key path for Reservations is ` meta / sales / reservations / <availabilityId> / <reservationId>`, while Availabilities are stored one level up, eg `meta / sales / reservations / <availabilityId> `, allowing all Reservations for an Availability to be queried (this is not currently needed, but may be useful when work to restore Availability size is implemented, more on this later). ### Lifecycle When a reservation is created, its size is deducted from the Availability, and when a reservation is deleted, any remaining size (bytes not written to disk) is returned to the Availability. If the request finishes, is cancelled (expired), or an error occurs, the Reservation is deleted (and any undownloaded bytes returned to the Availability). In addition, when the Sales module starts, any Reservations that are not actively being used in a filled slot, are deleted. Having a Reservation persisted until after a storage request is completed, will allow for the originally set Availability size to be reclaimed once a request contract has been completed. This is a feature that is yet to be implemented, however the work in this PR is a step in the direction towards enabling this. ### Unknowns Reservation size is determined by the `StorageAsk.slotSize`. If during download, more bytes than `slotSize` are attempted to be downloaded than this, then the Reservation update will fail, and the state machine will move to a `SaleErrored` state, deleting the Reservation. This will likely prevent the slot from being filled. ### Notes Based on #514
2023-09-29 04:33:08 +00:00
proc example*(_: type Reservation): Reservation =
Reservation.init(
availabilityId = AvailabilityId(array[32, byte].example),
size = uint16.example.u256,
slotId = SlotId.example
)
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
proc example*(_: type MerkleProof): MerkleProof =
MerkleProof.init(3, @[MultiHash.example]).tryget()
proc example*(_: type Poseidon2Proof): Poseidon2Proof =
var example = MerkleProof[Poseidon2Hash, PoseidonKeysEnum]()
example.index = 123
example.path = @[1, 2, 3, 4].mapIt( it.toF )
Wire sampler (#676) * Setting up testfixture for proof datasampler * Sets up calculating number of cells in a slot * Sets up tests for bitwise modulo * Implements cell index collection * setting up slot blocks module * Implements getting treeCID from slot * implements getting slot blocks by index * Implements out-of-range check for slot index * cleanup * Sets up getting sample from block * Implements selecting a cell sample from a block * Implements building a minitree for block cells * Adds method to get dataset block index from slot block index * It's running * splits up indexing * almost there * Fixes test. Implementation is now functional * Refactoring to object-oriented * Cleanup * Lining up output type with updated reference code. * setting up * Updates expected samples * Updates proof checking test to match new format * move builder to own dir * move sampler to own dir * fix paths * various changes to add support for the sampler * wip sampler implementation * don't use upraises * wip sampler integration * misc * move tests around * Various fixes to select correct slot and block index * removing old tests * cleanup * misc fix tests that work with correct cell indices * remove unused file * fixup logging * add logscope * truncate entropy to 31 bytes, otherwise it might be > than mod * forwar getCidAndProof to local store * misc * Adds missing test for initial-proving state * reverting back to correct slot/block indexing * fix tests for revert * misc * misc --------- Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-17 19:24:34 +00:00
example