* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* Rework AsyncIter
* Add tests for finishing iter on error
* Improved error handling for and additional tests
* Use new style of constructors
* Handle future cancellation
* Docs for constructors
* pulls in datastore-leveldb update
* bump
* Applies LevelDb as metadata store. Adds option for repostore.
* Sets submodule to main branch
* I can do syntax, me
* Removes wildcard from metadata query key
* Applies leveldb instead of sqlite-in-memory for tests
* Restores query key wildcard.
* Pins nim-datastore to latest master
* bumps leveldb to 0.1.4
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* json > nim-serde bump
Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.
* bump nim-serde
* change func to proc due to chronicles side effects
* import serde into utils/json, use as proxy
import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.
* update tests to use serde
* bump serde to latest
* remove testjson -- no longer needed
* bump serde in nimble
* updates to reconcile rebase with master
* Update codex-contracts-eth
* contracts: update G2Point definition
* integration: disable automatic advancing of time
reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds
* integration: fix parameters of marketplace payouts test
* integration: update test settings
* integration: fix typo
* integration: workaround for hardhat issue
Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.
* integration: remove origDatasetSizeInBlocks
* integration: fix proof parameters for test
* integration: do not log output by default
* integration: fix failure rate in test
* integration: fix warning
* integration: include clock in logs
* integration: allow for more periods
5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
Fixes validator, which uses transaction simulation to
check whether a proof can be marked as missing.
Fixes gas estimations, so they are done on the pending
block, instead of the latest block.
* rework backend to instantiate key at initialization
* add groth16 convertes for solidity
* prover taks num samples on construction
* add zkey file
* rework helpers
* rename types
* update tests
* reworked test helpers
* rename types
* rework test
* test all slots artifacts
* bump to latest version
* wire in circom backend
* should contain leafs
* adding circom compad and circuits deps
* update windows build
* fix windows build
* improve test names
* move proving defaults to codextypes
* remove unnedded inmports and move defaults to codextypes
* capture error code on backend failure
* Smart contracts update: Groth16Proof instead of bytes
* Use dummy verifier for now, until we can create ZK proofs
* Fix tests: submit proof only when slot is filled
* Submit dummy proofs for now
* More detailed log when proof submission failed
* Use dummy verifier for integration tests
For now at least
* Fix mistake in blanket renaming to ethProvider
* Update to latest codex-contracts-eth
* feat: zkey-hash from chain
* Fix zkeyHash
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* rework merkle tree support
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* use default hash codec
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* root returns a result
* import poseidon tests
* fix merge issues and cleanup a few warnings
* setting up slot builder
* Getting cids in slot
* ensures blocks are devisable by number of slots
* wip
* Implements indexing strategies
* Swaps in indexing strategy into erasure.
* wires slot and indexing tests up
* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges
* debugs indexing strategies
* Can select slot blocks
* finding number of pad cells
* Implements building slot tree
* finishes implementing slot builder
* Adds check that block size is a multiple of cell size
* Cleanup slotbuilder
* Review comments by Tomasz
* Fixes issue where ecK was used as numberOfSlots.
* rework merkle tree support
* deps
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* remove new codecs for now
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* fix bad `elements` iter
* bump
* bump
* wip
* reworking slotbuilder
* move out of manifest
* expose getCidAndProof
* import index strat...
* remove getMHash
* remove unused artifacts
* alias zero
* add digest for multihash
* merge issues
* remove unused hashes
* add option to result converter
* misc
* fix tests
* add helper to derive EC block count
* rename method
* misc
* bump
* extract slot root building into own proc
* revert to manifest to accessor
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* Workaround for Hardhat timestamp bug
Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.
This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.
Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.
* enable simulated proof failures for coverage
* comment out failing test on linux -- will be replaced
* bump codex contracts eth
* add back clock offset for non-hardhat cases
* bump codex-contracts-eth
increases pointer by 67 blocks each period increase
* Add `codex_use_hardhat` flag to coverage tests
* Adds test for encoding/decoding protected manifest
* Setting up verifiable manifest
* mysterious mysteries
* Successful encoding test for verifiable manifests
* extracts toF out of users of manifest code
* Update codex/manifest/coders.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
* Review comments by Dmitriy
* Adds missing verifiable print to $ method.
* Replace poseidon2 F type with int as temporary stand-in for verification hashes
* Replaces verification hash placeholder with CID
---------
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* [build] add nim-poseidon2
* [build] Workaround incompatibility constantine and secp256k1
Constantine sets compiler flag that is incompatible
with the assembly code in secp256k1:
5f7ba18f2e/constantine/platforms/isa/macro_assembler_x86.nim (L19)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* [docs] fix two client scenario: add missing collateral
* [integration] separate step to wait for node to be started
* [cli] add option to specify ethereum private key
* Remove unused imports
* Fix warnings
* [integration] move type definitions to correct place
* [integration] wait a bit longer for a node to start in debug mode
When e.g. running against Taiko test net rpc, the node start
takes longer
* [integration] simplify handling of codex node and client
* [integration] add Taiko integration test
* [contracts] await token approval confirmation before next tx
* [contracts] deployment address of marketplace on Taiko
* [cli] --eth-private-key now takes a file name
Instead of supplying the private key on the command line,
expect the private key to be in a file with the correct
permissions.
* [utils] Fixes undeclared `activeChroniclesStream` on Windows
* [build] update nim-ethers to include PR #52
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [cli] Better error messages when reading eth private key
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [integration] simplify reading of cmd line arguments
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* [build] update to latest version of nim-ethers
* [contracts] updated contract address for Taiko L2
* [build] update codex contracts to latest version
---------
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).
Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.
When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.
Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used
The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).
When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.
SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.
Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.
Prevent loading request from contract (network request) if there is an existing item in queue for that request.
Check availability before adding request to queue.
Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).
#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop
Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).
Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
- Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks
#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.
### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
- prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating
An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.
Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.
Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
- remove unneeded error handling as pragmas were moved
Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.
Sales tests: shut down repo after sales stop, to fix SIGABRT in CI
### Add async Promise API
- modelled after JavaScript Promise API
- alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
- Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
- Add cancellation callbacks to utils/then, ensuring cancellations are handled properly
## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb
---------
Co-authored-by: Jaremy Creechley <creechley@gmail.com>
* Support for building docker images with local modifications for the purpose of testing and debugging
* exposes peer information via debug/info
* api-names slightly kinder to json serializers
* Moves debug image-building
* fixes misalignment of debug peer info array
* Changes switchPeers source from KeyBook to AddressBook (filed ticket in libp2p, discussed with Tanguy)
* Limited success with dist-test peer discovery tests
* Removes unnecessary random-timer
* bumps dht repo. Adds peerId to formatNode
* Removes unused prints
* bumps libp2p-dht
* Exposes node address on debug api
* Adds traces
* review comments by me
* Hides debug/peers api behind compile flag
* Waiting for nim-libp2p-dht PR merge
* bumps libp2p-dht back to main after PRs were merged there.
* Cleanup
* [contracts] Add SlotFreed event
* [integration] allow test node to be stopped twice
* [cli] add --validator option
* [contracts] remove dead code
* [contracts] instantiate OnChainMarket and OnChainClock only once
* [contracts] add Validation
* [sales] remove duplicate import
* [market] add missing import
* [market] subscribe to all SlotFilled events
* [market] add freeSlot()
* [sales] fix warnings
* [market] subscribe to SlotFreed events
* [contracts] fix warning
* [validator] keep track of filled slots
* [validation] remove slots that have ended
* [proving] absorb Proofs into Market
Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.
* [market] add markProofAsMissing()
* [clock] speed up waiting for clock in tests
* [validator] mark proofs as missing
* [timer] fix error on node shutdown
* [cli] handle --persistence and --validator separately
* [market] allow retrieval of proof timeout value
* [validator] do not subscribe to SlotFreed events
Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.
* [validator] Start validation at the start of a new period
To decrease the likelihood that we hit the validation timeout.
* [validator] do not mark proofs as missing after timeout
* [market] check whether proof can be marked as missing
* [validator] simplify validation
Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.
* [build] use nim-ethers PR #40
Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.
https://github.com/status-im/nim-ethers/pull/40
* [integration] integration test for validator
* [validator] monitor a maximum number of slots
Adds cli parameter --validator-max-slots.
* [market] fix missing collateral argument
After rebasing, add the new argument to fillSlot calls.
* [build] update to nim-ethers 0.2.5
* [validator] use Set instead of Table to keep track of slots
* [validator] add logging
* [validator] add test for slot failure
* [market] use "pending" blocktag to use more up to date block time
* [contracts] remove unused import
* [validator] fix: wait until after period ends
The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
Eric
Giuliano Mega
Tomasz Bekas
Ben Bierens
Adam Uhlíř
markspanbroek
Dmitriy Ryajov
Jaremy Creechley