logos-storage-nim/codex/utils.nim

import ./utils/asyncheapqueue
import ./utils/fileutils

export asyncheapqueue, fileutils


func divUp*[T](a, b : T): T =
  ## Division with result rounded up (rather than truncated as in 'div')
  assert(b != 0)
  if a==0:  0  else:  ((a - 1) div b) + 1

func roundUp*[T](a, b : T): T =
  ## Round up 'a' to the next value divisible by 'b'
  divUp(a,b) * b
Engine (#89) * rework discovery with async queues * increase max message size for large manifests * increase sleep time to 100 millis * pass config * check for nil on start/stop * fix tests and split out discovery tests * don't auto mount network * add discovery tests * rework moc discovery * move discovery moc to disc dir * don't force logging syncs * don't force moc discovery on all tests * rework discovery with methods * add top level utils file * don't use asyncCheck * don't pass entire blocks to list blocks calback * spelling * - don't send want reqs to peers reporting the cid - Don't request blocks directly on presense update, use `requestBlock` * bug, nodes should not have blocks in local store * Add failing test * prefetch blocks so that download isn't serial * if request already pending, return the handle * fire discovery if no peers report block as have * only query discovery if not enough nodes for cid * wrap async req in template * use non awaiting version of queue routines * rework E2E tests as unittest * re-add chronicles sinks Co-authored-by: Tanguy <tanguy@status.im> 2022-05-12 15:52:03 -06:00			`import ./utils/asyncheapqueue`
			`import ./utils/fileutils`

Upload authenticators (#108) * initial implementation of storage proofs upload * make sure proof verifies with after deserializing * add por store * rename por store to stp store * rename porstore to stpstore * add support for host discovery to discovery mock * add tags upload network tests 2022-05-25 20:29:31 -06:00			`export asyncheapqueue, fileutils`
Download files without padding (#218) The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations. Note that the wire format of Manifest was changed, so we need to recreate all BlockStores. * Download without padding * Fixed chunker tests * Chunker: get rid of RabinChunker * Verify offset in the chunker tests * Use manifest.originalBytesPadded in StoreStream.size * StoreStream: replace emptyBlock with zeroMem * Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return * Manifest: verify originalBytes and originalLen on new/encode/decode Also set originalBytes in each Manifest creation/update scenario * Manifest: comments, split code into sections * Reordered parameters to deal with int64 size in 32-bit builds * TestNode.nim: combine Store and Retrieve tests 1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data 2. New test compares only file contents, manifest contents considered an implementation detail 3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch * TestNode.nim: code refactoring * Manifest.add: one more test * Manifest.verify: return Result instead of raising Defect * Node.store: added blockSize parameter 2022-08-24 15:15:59 +03:00

			`func divUp*[T](a, b : T): T =`
			`## Division with result rounded up (rather than truncated as in 'div')`
			`assert(b != 0)`
			`if a==0: 0 else: ((a - 1) div b) + 1`

			`func roundUp*[T](a, b : T): T =`
			`## Round up 'a' to the next value divisible by 'b'`
			`divUp(a,b) * b`