* nix: make derivation and update shell
Create a structure for nix files. Add the derivation file which is using
system Nim to compile Codex.
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940
Signed-off-by: markoburcul <marko@status.im>
* nim-circom-compat: update
Include commit which allows building circom-compat-ffi using Nix(doesn't
affect current usage of the submodule).
Referenced issue: https://github.com/codex-storage/nim-codex/issues/940
Signed-off-by: markoburcul <marko@status.im>
* makefile: fix for detecting linux arch
Signed-off-by: markoburcul <marko@status.im>
---------
Signed-off-by: markoburcul <marko@status.im>
* Use http subscriptions instead of websocket for tests
To work around this issue when subscriptions are
inactive for more than 5 minutes:
https://github.com/NomicFoundation/hardhat/issues/2053
Use 100 millisecond polling; default polling interval
of 4 seconds is too close to the 5 second timeout for
`check eventually`.
* use .confirm(1) instead of confirm(0)
confirm(0) doesn't wait at all, confirm(1) waits
for the transaction to be mined
* speed up partial payout integration test
* update nim-ethers to version 0.10.0
includes fixes for http polling and .confirm()
* fix timing of marketplace tests
allow for a bit more time to withdraw funds
* use .confirm(1) in marketplace tests
to ensure that the transaction has been processed
before continuing with the test
* fix timing issue in validation unit test
* fix proof integration test
there were two logic errors in this test:
- a slot is freed anyway at the end of the contract
- when starting the request takes a long time, the
first slot can already be freed because there were
too many missing proofs
* fix intermittent error in contract tests
currentTime() doesn't always correctly reflect
the time of the next transaction
* reduce number of slots in integration test
otherwise the windows runner in the CI won't
be able to start the request before it expires
* fix timing in purchasing test
allow for a bit more time for a request to
be submitted
* fix timing of request submission in test
windows ci is so slow, it can take up to 40 seconds
just to submit a storage request to hardhat
* increase proof period to 90 seconds
* adjust timing of integration tests
reason: with the increased period length of 90 seconds, it
can take longer to wait for a stable challenge at the
beginning of a period.
* increase CI timeout to 2 hours
* Fix slow builds on windows
apparently it takes windows 2-3 seconds to
resolve "localhost" to 127.0.0.1 for every
json-rpc connection that we make 🤦
* fix: bumps ethers to fix missing nonce error
* fix was merged in nim-ethers
---------
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
* chore: bump dependencies, including nim-ethers with chronos v4 support
Bumps the following dependencies:
- nim-ethers to commit 507ac6a4cc71cec9be7693fa393db4a49b52baf9 which contains a pinned nim-eth version. This is to be replaced by a versioned library, so it will be pinned to a particular version. There is a crucial fix in this version of ethers that fixes nonce management which is causing issues in the Codex testnet.
- nim-json-rpc to v0.4.4
- nim-json-serialization to v0.2.8
- nim-serde to v1.2.2
- nim-serialization to v0.2.4
Currently, one of the integration tests is failing.
* fix integration test
- When a state's run was cancelled, it was being caught as an error due to catching all CatchableErrors. This caused a state transition to SaleErrored, however cancellation of run was not actually an error. Handling this correctly fixed the issue.
- Stopping of the clock was moved to after `HostInteractions` (sales) which avoided an assertion around getting time when the clock was not started.
* bump ethers to include nonce fix and filter not found fix
* bump ethers: fixes missing symbol not exported in ethers
* Fix cirdl test imports/exports
* Debugging in ci
* Handle CancelledErrors for state.run in one place only
* Rename `config` to `configuration`
There was a symbol clash preventing compilation and it was easiest to rename `config` to `configuration` in the contracts. Not even remotely ideal, but it was the only way.
* bump ethers to latest
Prevents an issue were `JsonNode.items` symbol could not be found
* More changes to support `config` > `configuration`
* cleanup
* testing to see if this fixes failure in ci
* bumps contracts
- ensures slot is free before allowing reservation
- renames config to configuration to avoid symbol clash
* convert EthersError to MarketError
* change `canReserveSlot` and `reserveSlot` parameters
Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.
* Add SaleSlotReserving
Adds a new state, SaleSlotReserving, that attempts to reserve a slot before downloading.
If the slot cannot be reserved, the state moves to SaleIgnored.
On error, the state moves to SaleErrored.
SaleIgnored is also updated to pass in `reprocessSlot` and `returnBytes`, controlling the behaviour in the Sales module after the slot is ignored. This is because previously it was assumed that SaleIgnored was only reached when there was no Availability. This is no longer the case, since SaleIgnored can now be reached when a slot cannot be reserved.
* Update SalePreparing
Specify `reprocessSlot` and `returnBytes` when moving to `SaleIgnored` from `SalePreparing`.
Update tests to include test for a raised CatchableError.
* Fix unit test
* Modify `canReserveSlot` and `reverseSlot` params after rebase
* Update MockMarket with new `canReserveSlot` and `reserveSlot` params
* fix after rebase
also bump codex-contracts-eth to master
* feat(slot-reservations): Support reserving slots
Closes#898.
Wire up reserveSlot and canReserveSlot contract calls, but don't call them
* Remove return value from `reserveSlot`
* convert EthersError to MarketError
* Move convertEthersError to reserveSlot
* bump codex-contracts-eth after rebase
* change `canReserveSlot` and `reserveSlot` parameters
Parameters for `canReserveSlot` and `reserveSlot` were changed from `SlotId` to `RequestId` and `UInt256 slotIndex`.
* bump codex-contracts-eth after rebase
* bump codex-contracts-eth to master after codex-contracts-eth/pull/177 merged
* Introduces a start method to prover
* Moves backend creation into start method
* sets up three paths for backend initialization
* Extracts backend initialization to backend-factory
* Implements loading backend from cli files or previously downloaded local files
* Wires up downloading and unzipping
* functional implementation
* Fixes testprover.nim
* Sets up tests for backendfactory
* includes libzip-dev
* pulls in updated contracts
* removes integration cli tests for r1cs, wasm, and zkey file arguments.
* Fixes issue where inner-scope values are lost before returning
* sets local proof verification for dist-test images
* Adds two traces and bumps nim-ethers
* Adds separate path for circuit files
* Create circuit dir if not exists
* fix: make sure requestStorage is mined
* fix: correct place to plug confirm
* test: fixing contracts tests
* Restores gitmodules
* restores nim-datastore reference
* Sets up downloader exe
* sets up tool skeleton
* implements getting of circuit hash
* Implements downloader tool
* sets up test skeleton
* Implements test for cirdl
* includes testTools in testAll
* Cleanup building.md
* cleans up previous downloader implementation
* cleans up testbackendfactory
* moves start of prover into node.nim
* Fills in arguments in example command
* Initializes backend in prover constructor
* Restores tests
* Restores tests for cli instructions
* Review comments by Dmitriy, part 1
* Quotes path in download instruction.
* replaces curl with chronos http session
* Moves cirdl build output to 'build' folder.
* Fixes chronicles log output
* Add cirdl support to the codex Dockerfile
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the docker entrypoint
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Add cirdl support to the release workflow
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Disable verify_circuit flag for releases
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
* Removes backendFactory placeholder type
* wip
* Replaces zip library with status-im/zippy library (which supports zip and tar)
* Updates cirdl to not change circuitdir folder
* Switches from zip to tar.gz
* Review comments by Dmitriy
* updates codex-contracts-eth
* Adds testTools to CI
* Adds check for access to config.circuitdir
* Update fixture circuit zkey
* Update matrix to run tools tests on Windows
* Adds 'deps' dependency for cirdl
* Adjust docker-entrypoint.sh to use CODEX_CIRCUIT_DIR env var
* Review comments by Giuliano
---------
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
Co-authored-by: Veaceslav Doina <20563034+veaceslavdoina@users.noreply.github.com>
* feat: add `--payout-address`
Allows SPs to be paid out to a separate address, keeping their profits secure.
Supports https://github.com/codex-storage/codex-contracts-eth/pull/144 in the nim-codex client.
* Remove optional payoutAddress
Change --payout-address so that it is no longer optional. There is no longer an overload in `Marketplace.sol` for `fillSlot` accepting no `payoutAddress`.
* Update integration tests to include --payout-address
* move payoutAddress from fillSlot to freeSlot
* Update integration tests to use required payoutAddress
- to make payoutAddress required, the integration tests needed to avoid building the cli params until just before starting the node, otherwise if cli params were added ad-hoc, there would be an error after a non-required parameter was added before a required parameter.
* support client payout address
- withdrawFunds requires a withdrawAddress parameter, directs payouts for withdrawing of client funds (for a cancelled request) to go to that address.
* fix integration test
adds --payout-address to validators
* refactor: support withdrawFunds and freeSlot optional parameters
- withdrawFunds has an optional parameter for withdrawRecipient
- freeSlot has optional parameters for rewardRecipient and collateralRecipient
- change --payout-address to --reward-recipient to match contract signature naming
* Revert "Update integration tests to include --payout-address"
This reverts commit 8f9535cf35b0f2b183ac4013a7ed11b246486964.
There are some valid improvements to the integration tests, but they can be handled in a separate PR.
* small fix
* bump contracts to fix marketplace spec
* bump codex-contracts-eth, now rebased on master
* bump codex-contracts-eth
now that feat/reward-address has been merged to master
* clean up, comments
* add changes to use chronos v4 in compat mode
* switch chronos to compat fix branch
* use nimbus-build-system with configurable Nim repo
* add missing imports
* add missing await
* bump compat
* pin nim version in Makefile
* add await instead of asyncSpawn to advertisement queue loop
* bump DHT to v0.5.0
* allow error state of `onBatch` to propagate upwards in test code
* pin Nim compiler commit to avoid fetching stale branch
* make CI build against branch head instead of merge
* fix handling of return values in testslotqueue
* add logs and test
* add Merkle proof checks
* factor out Circom input normalization, fix proof input serialization
* add test and update existing ones
* update circuit assets
* add back trace message
* switch contracts to fix branch
* update codex-contracts-eth to latest
* do not expose prove with prenormalized inputs
* Rework AsyncIter
* Add tests for finishing iter on error
* Improved error handling for and additional tests
* Use new style of constructors
* Handle future cancellation
* Docs for constructors
* pulls in datastore-leveldb update
* bump
* Applies LevelDb as metadata store. Adds option for repostore.
* Sets submodule to main branch
* I can do syntax, me
* Removes wildcard from metadata query key
* Applies leveldb instead of sqlite-in-memory for tests
* Restores query key wildcard.
* Pins nim-datastore to latest master
* bumps leveldb to 0.1.4
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* json > nim-serde bump
Should wait until serde is integrated into nim-ethers before making these changes as there will be less import exceptions required.
* bump nim-serde
* change func to proc due to chronicles side effects
* import serde into utils/json, use as proxy
import nim-serde into utils/json and use utils/json as a proxy for serde functions, including overloading `%` and `fromJson` for application types.
* update tests to use serde
* bump serde to latest
* remove testjson -- no longer needed
* bump serde in nimble
* updates to reconcile rebase with master
* Update codex-contracts-eth
* contracts: update G2Point definition
* integration: disable automatic advancing of time
reason: makes reasoning about timing in tests harder,
because the period is set to 60 seconds in the
marketplace configuration, but this code switches to
a new period every 500 milliseconds
* integration: fix parameters of marketplace payouts test
* integration: update test settings
* integration: fix typo
* integration: workaround for hardhat issue
Subscriptions expire after 5 minutes when using
websockets. Use http and polling instead.
* integration: remove origDatasetSizeInBlocks
* integration: fix proof parameters for test
* integration: do not log output by default
* integration: fix failure rate in test
* integration: fix warning
* integration: include clock in logs
* integration: allow for more periods
5 periods was cutting it close, if we get too much
pointer downtime, then the test would fail
Fixes validator, which uses transaction simulation to
check whether a proof can be marked as missing.
Fixes gas estimations, so they are done on the pending
block, instead of the latest block.
* rework backend to instantiate key at initialization
* add groth16 convertes for solidity
* prover taks num samples on construction
* add zkey file
* rework helpers
* rename types
* update tests
* reworked test helpers
* rename types
* rework test
* test all slots artifacts
* bump to latest version
* wire in circom backend
* should contain leafs
* adding circom compad and circuits deps
* update windows build
* fix windows build
* improve test names
* move proving defaults to codextypes
* remove unnedded inmports and move defaults to codextypes
* capture error code on backend failure
* Smart contracts update: Groth16Proof instead of bytes
* Use dummy verifier for now, until we can create ZK proofs
* Fix tests: submit proof only when slot is filled
* Submit dummy proofs for now
* More detailed log when proof submission failed
* Use dummy verifier for integration tests
For now at least
* Fix mistake in blanket renaming to ethProvider
* Update to latest codex-contracts-eth
* feat: zkey-hash from chain
* Fix zkeyHash
---------
Co-authored-by: Adam Uhlíř <adam@uhlir.dev>
* implement a logging proxy
The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks
* Rename logging -> logutils to avoid ambiguity with common names
* clean up
* add setProperty for JsonRecord, remove nim-json-serialization conflict
* Allow specifying textlines and json format separately
Not specifying a LogFormat will apply the formatting to both textlines and json sinks.
Specifying a LogFormat will apply the formatting to only that sink.
* remove unneeded usages of std/json
We only need to import utils/json instead of std/json
* move serialization from rest/json to utils/json so it can be shared
* fix NoColors ambiguity
Was causing unit tests to fail on Windows.
* Remove nre usage to fix Windows error
Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)
* Add logutils module doc
* Shorten logutils.formatIt for `NBytes`
Both json and textlines formatIt were not needed, and could be combined into one formatIt
* remove debug integration test config
debug output and logformat of json for integration test logs
* Use ## module doc to support docgen
* bump nim-poseidon2 to export fromBytes
Before the changes in this branch, fromBytes was likely being resolved by nim-stew, or other dependency. With the changes in this branch, that dependency was removed and fromBytes could no longer be resolved. By exporting fromBytes from nim-poseidon, the correct resolution is now happening.
* fixes to get compiling after rebasing master
* Add support for Result types being logged using formatIt
* rework merkle tree support
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* use default hash codec
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* root returns a result
* import poseidon tests
* fix merge issues and cleanup a few warnings
* setting up slot builder
* Getting cids in slot
* ensures blocks are devisable by number of slots
* wip
* Implements indexing strategies
* Swaps in indexing strategy into erasure.
* wires slot and indexing tests up
* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges
* debugs indexing strategies
* Can select slot blocks
* finding number of pad cells
* Implements building slot tree
* finishes implementing slot builder
* Adds check that block size is a multiple of cell size
* Cleanup slotbuilder
* Review comments by Tomasz
* Fixes issue where ecK was used as numberOfSlots.
* rework merkle tree support
* deps
* rename merkletree -> codexmerkletree
* treed and proof encoding/decoding
* style
* adding codex merkle and coders tests
* remove new codecs for now
* proof size changed
* add from nodes test
* shorte file names
* wip poseidon tree
* shorten file names
* fix bad `elements` iter
* bump
* bump
* wip
* reworking slotbuilder
* move out of manifest
* expose getCidAndProof
* import index strat...
* remove getMHash
* remove unused artifacts
* alias zero
* add digest for multihash
* merge issues
* remove unused hashes
* add option to result converter
* misc
* fix tests
* add helper to derive EC block count
* rename method
* misc
* bump
* extract slot root building into own proc
* revert to manifest to accessor
---------
Co-authored-by: benbierens <thatbenbierens@gmail.com>
* Workaround for Hardhat timestamp bug
Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.
This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.
Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.
* enable simulated proof failures for coverage
* comment out failing test on linux -- will be replaced
* bump codex contracts eth
* add back clock offset for non-hardhat cases
* bump codex-contracts-eth
increases pointer by 67 blocks each period increase
* Add `codex_use_hardhat` flag to coverage tests
* Adds test for encoding/decoding protected manifest
* Setting up verifiable manifest
* mysterious mysteries
* Successful encoding test for verifiable manifests
* extracts toF out of users of manifest code
* Update codex/manifest/coders.nim
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
* Review comments by Dmitriy
* Adds missing verifiable print to $ method.
* Replace poseidon2 F type with int as temporary stand-in for verification hashes
* Replaces verification hash placeholder with CID
---------
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
* [build] add nim-poseidon2
* [build] Workaround incompatibility constantine and secp256k1
Constantine sets compiler flag that is incompatible
with the assembly code in secp256k1:
5f7ba18f2e/constantine/platforms/isa/macro_assembler_x86.nim (L19)
---------
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Marko Burčul
Adam Uhlíř
Ben Bierens
markspanbroek
Eric
Giuliano Mega
Tomasz Bekas
Dmitriy Ryajov