Implements delta-based WantList updates to reduce network traffic during
block exchange. Only sends newly added blocks instead of resending the
entire WantList on every refresh.
Also some network related fixes:
- Add TCP_NODELAY flag to prevent Nagle's algorithm delays
- Clear sendConn on stream reset to allow garbage collection
- Improve error handling in NetworkPeer.send()
Part of https://github.com/codex-storage/nim-codex/issues/974
Signed-off-by: Chrysostomos Nanakos <chris@include.gr>
* cleanup imports and logs
* add BlockHandle type
* revert deps
* refactor: async error handling and future tracking improvements
- Update async procedures to use explicit raises annotation
- Modify TrackedFutures to handle futures with no raised exceptions
- Replace `asyncSpawn` with explicit future tracking
- Update test suites to use `unittest2`
- Standardize error handling across network and async components
- Remove deprecated error handling patterns
This commit introduces a more robust approach to async error handling and future management, improving type safety and reducing potential runtime errors.
* bump nim-serde
* remove asyncSpawn
* rework background downloads and prefetch
* imporove logging
* refactor: enhance async procedures with error handling and raise annotations
* misc cleanup
* misc
* refactor: implement allFinishedFailed to aggregate future results with success and failure tracking
* refactor: update error handling in reader procedures to raise ChunkerError and CancelledError
* refactor: improve error handling in wantListHandler and accountHandler procedures
* refactor: simplify LPStreamReadError creation by consolidating parameters
* refactor: enhance error handling in AsyncStreamWrapper to catch unexpected errors
* refactor: enhance error handling in advertiser and discovery loops to improve resilience
* misc
* refactor: improve code structure and readability
* remove cancellation from addSlotToQueue
* refactor: add assertion for unexpected errors in local store checks
* refactor: prevent tracking of finished futures and improve test assertions
* refactor: improve error handling in local store checks
* remove usage of msgDetail
* feat: add initial implementation of discovery engine and related components
* refactor: improve task scheduling logic by removing unnecessary break statement
* break after scheduling a task
* make taskHandler cancelable
* refactor: update async handlers to raise CancelledError
* refactor(advertiser): streamline error handling and improve task flow in advertise loops
* fix: correct spelling of "divisible" in error messages and comments
* refactor(discovery): simplify discovery task loop and improve error handling
* refactor(engine): filter peers before processing in cancelBlocks procedure
* adding basic retry functionality
* avoid duplicate requests and batch them
* fix cancelling blocks
* properly resolve blocks
* minor cleanup - use `self`
* avoid useless asyncSpawn
* track retries
* limit max inflight and set libp2p maxIncomingStreams
* cleanup
* add basic yield in readLoop
* use tuple instead of object
* cleanup imports and logs
* increase defaults
* wip
* fix prefetch batching
* cleanup
* decrease timeouts to speedup tests
* remove outdated test
* add retry tests
* should track retries
* remove useless test
* use correct block address (index was off by 1)
* remove duplicate noop proc
* add BlockHandle type
* Use BlockHandle type
* add fetchLocal to control batching from local store
* add format target
* revert deps
* adjust quotaMaxBytes
* cleanup imports and logs
* revert deps
* cleanup blocks on cancelled
* terminate erasure and prefetch jobs on stream end
* split storing and retrieving data into separate tests
* track `b.discoveryLoop` future
* misc
* remove useless check
* integration: move REST API tests into their own module
* integration: move upload and download tests into their own module
* integration: move purchasing tests into their own module
* integration: move marketplace tests to the right module
* integration: mine a block *after* starting nodes
To ensure that tests involving multiple nodes do
not start with out-of-sync clocks
* Fix: do not swallow CancellationErrors
* integration: avoid underflow in UInt256
* network: remove unnecessary error handling
No Exceptions can occur, only Defects, because everything
is asyncSpawned.
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* network: do not raise in asyncSpawned proc
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Updates logging for file upload
* Restores trace for placing block and proof in repo store
* Reduces logging while transmitting blocks
* unnecessary formatter
* Clean up some more download related traces
* much better
* Review comment by dryajov
* add block cancellation support + tests
* tie issueCancellations into resolveBlocks for proper exception tracking, address comments
* pull cancellation as separate primitive in BlockExcNetwork
* use allFutures, rename issueBlockCancellations -> cancelBlocks
* use trc instead of wrn to register send error
* do not log peer IDs
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* Blockexchange uses merkle root and index to fetch blocks
* Links the network store getTree to the local store.
* Update codex/stores/repostore.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
* Rework erasure.nim to include recent cleanup
* Revert accidential changes to lib versions
* Addressing review comments
* Storing proofs instead of trees
* Fix a comment
* Fix broken tests
* Fix for broken testerasure.nim
* Addressing PR comments
---------
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* Fixes/workarounds for nimsuggest failures in codex.nim.
* remove rng prefix - it appears to work now
* format new's to be more consistent
* making proc formatting a bit more consistent
* applying styleCheck
* stuck on vendor folder
* Applies style check
* Turns styleCheck back off
* switches to stylecheck:usages
* Fixes empty template casing
* rolls up nim-blscurve, nim-datastore, nim-ethers, nim-leopard, and nim-taskpools.
* bumps nim-confutils and removes unused import from fileutils.nim
* Unused using in fileutils.nim is required by CI
* Reverts bump of nim-confutils module
* initial implementation of storage proofs upload
* make sure proof verifies with after deserializing
* add por store
* rename por store to stp store
* rename porstore to stpstore
* add support for host discovery to discovery mock
* add tags upload network tests
