* don't unroll wantCids when sending wantBlock message in blockPresenceHandler
* workaround logging
* Fixes logformatting upraises for sequences.
* Applies upraises rule for setProperty of textmode for sequences.
* Replaces upraises with raises
* Removes redundant log in sendWantHave
* Updates logging for file upload
* Restores trace for placing block and proof in repo store
* Reduces logging while transmitting blocks
* unnecessary formatter
* Clean up some more download related traces
* much better
* Review comment by dryajov
* wire prover into node
* stricter case object checks
* return correct proof
* misc renames
* adding usefull traces
* fix nodes and tolerance to match expected params
* format challenges in logs
* add circom compat to solidity groth16 convertion
* update
* bump time to give nodes time to load with all circom artifacts
* misc
* misc
* use correct dataset geometry in erasure
* make errors more searchable
* use parens around `=? (await...)` calls
* styling
* styling
* use push raises
* fix to match constructor arguments
* merge master
* merge master
* integration: fix proof parameters for a test
Increased times due to ZK proof generation.
Increased storage requirement because we're now hosting
5 slots instead of 1.
* sales: calculate initial proof at start of period
reason: this ensures that the period (and therefore
the challenge) doesn't change while we're calculating
the proof
* integration: fix proof parameters for tests
Increased times due to waiting on next period.
Fixed data to be of right size.
Updated expected payout due to hosting 5 slots.
* sales: wait for stable proof challenge
When the block pointer is nearing the
wrap-around point, we wait another period
before calculating a proof.
* fix merge conflict
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* Applies peer-scoped lock to peer task handler.
* Replace async lock with delete-first approach.
* Cleanup some logging
* Adds inFlight flag to WantListEntry
* Clears inflight flag when local retrieval fails.
* Adds test for setting of in-flight
* Adds test for clearing in-flight when lookup fails
* Review comments by Tomasz
---------
Co-authored-by: gmega <giuliano.mega@gmail.com>
* add block cancellation support + tests
* tie issueCancellations into resolveBlocks for proper exception tracking, address comments
* pull cancellation as separate primitive in BlockExcNetwork
* use allFutures, rename issueBlockCancellations -> cancelBlocks
* use trc instead of wrn to register send error
* do not log peer IDs
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* Setting up testfixture for proof datasampler
* Sets up calculating number of cells in a slot
* Sets up tests for bitwise modulo
* Implements cell index collection
* setting up slot blocks module
* Implements getting treeCID from slot
* implements getting slot blocks by index
* Implements out-of-range check for slot index
* cleanup
* Sets up getting sample from block
* Implements selecting a cell sample from a block
* Implements building a minitree for block cells
* Adds method to get dataset block index from slot block index
* It's running
* splits up indexing
* almost there
* Fixes test. Implementation is now functional
* Refactoring to object-oriented
* Cleanup
* Lining up output type with updated reference code.
* setting up
* Updates expected samples
* Updates proof checking test to match new format
* move builder to own dir
* move sampler to own dir
* fix paths
* various changes to add support for the sampler
* wip sampler implementation
* don't use upraises
* wip sampler integration
* misc
* move tests around
* Various fixes to select correct slot and block index
* removing old tests
* cleanup
* misc
fix tests that work with correct cell indices
* remove unused file
* fixup logging
* add logscope
* truncate entropy to 31 bytes, otherwise it might be > than mod
* forwar getCidAndProof to local store
* misc
* Adds missing test for initial-proving state
* reverting back to correct slot/block indexing
* fix tests for revert
* misc
* misc
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* rework merkle tree support
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* use default hash codec
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* root returns a result
* import poseidon tests
* fix merge issues and cleanup a few warnings
* setting up slot builder
* Getting cids in slot
* ensures blocks are devisable by number of slots
* wip
* Implements indexing strategies
* Swaps in indexing strategy into erasure.
* wires slot and indexing tests up
* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges
* debugs indexing strategies
* Can select slot blocks
* finding number of pad cells
* Implements building slot tree
* finishes implementing slot builder
* Adds check that block size is a multiple of cell size
* Cleanup slotbuilder
* Review comments by Tomasz
* Fixes issue where ecK was used as numberOfSlots.
* rework merkle tree support
* deps
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* remove new codecs for now
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* fix bad `elements` iter
* bump
* bump
* wip
* reworking slotbuilder
* move out of manifest
* expose getCidAndProof
* import index strat...
* remove getMHash
* remove unused artifacts
* alias zero
* add digest for multihash
* merge issues
* remove unused hashes
* add option to result converter
* misc
* fix tests
* add helper to derive EC block count
* rename method
* misc
* bump
* extract slot root building into own proc
* revert to manifest to accessor
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* Blockexchange uses merkle root and index to fetch blocks
* Links the network store getTree to the local store.
* Update codex/stores/repostore.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
* Rework erasure.nim to include recent cleanup
* Revert accidential changes to lib versions
* Addressing review comments
* Storing proofs instead of trees
* Fix a comment
* Fix broken tests
* Fix for broken testerasure.nim
* Addressing PR comments
---------
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Adds metrics to block exchange.
* Adds metrics to purchasing
* Adds metrics to upload and download API
* Adds metrics to the repo store
* Fixes exception in repostore start.
* Merge managed to mess up indentation.
* Adds sequence diagram for local download
* Adds network download flow
* makes node properly request block when presence-list is received
* starting to work :O
* Fixes fully connected download test up to 20 nodes.
* Better sequence diagram of network block resolution
* Cleans up comments and traces.
* Accidentally deleted important else
* Review comments by Dmitriy
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* Fixes/workarounds for nimsuggest failures in codex.nim.
* remove rng prefix - it appears to work now
* format new's to be more consistent
* making proc formatting a bit more consistent
* applying styleCheck
* stuck on vendor folder
* Applies style check
* Turns styleCheck back off
* switches to stylecheck:usages
* Fixes empty template casing
* rolls up nim-blscurve, nim-datastore, nim-ethers, nim-leopard, and nim-taskpools.
* bumps nim-confutils and removes unused import from fileutils.nim
* Unused using in fileutils.nim is required by CI
* Reverts bump of nim-confutils module
* [build] disable XCannotRaiseY hint
There are too many {.raises:[Defect].} in the
libraries that we use, drowning out all other
warnings and hints
* [build] disable BareExcept warning
Not yet enabled in a released version of Nim,
so libraries that we depend on have not fixed
this yet, drowning out our own hints and warnings
* [build] disable DotLikeOps warning
dot-like ops were an experiment that is not going
land in Nim
* [build] compile log statements in tests
When running tests, all log statements are compiled.
They are filtered out at runtime during a test run.
* [build] do not build executable when running unit test
It's already built in the integration test
* [build] Fix warnings
- remove unused code
- remove unused imports
- stop using deprecated stuff
* [build] Put compiler flags behind nim version checks
* [CI] remove Nim 1.2 compatibility
* add format for cid
* cid formatIt change
* track nim-libp2p-unstable
* rework probuf serialization for por
* add missing include
* removing nim protobuf serialization
* rollback to dht to main
* remove protobuf serialization import
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`
use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors
also make some logic and error handling/messages more consistent across
BlockStore implementations
closes#177closes#182closes#210
alternative to #205, #209
BlockStore API got new return types (rationale in https://github.com/status-im/nim-codex/issues/123#issuecomment-1163797753):
- getBlock: Future[?! (?Block)]
- putBlock/delBlock/listBlocks: Future[?!void]
- hasBlock: Future[?!bool]
Plus refactored readOnce(StoreStream) and check received data in its tests.
And replaced local use of AsyncHeapQueue with seq.sort.
Ben Bierens
Giuliano Mega
markspanbroek
Eric
Dmitriy Ryajov
Tomasz Bekas
Slava
Jaremy Creechley
jessiebroke
Michael Bradley
Bulat-Ziganshin