Commit Graph

306 Commits

Author SHA1 Message Date
Eric b186afbc4f
bump ethers, remove upraises from callbacks (#664) 2024-01-09 15:51:34 +11:00
Dmitriy Ryajov fffb674bba
Integrate slot builder (#666)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* fix merge issues and cleanup a few warnings

* setting up slot builder

* Getting cids in slot

* ensures blocks are devisable by number of slots

* wip

* Implements indexing strategies

* Swaps in indexing strategy into erasure.

* wires slot and indexing tests up

* Fixes issue where indexing strategy stepped gives wrong values for smallest of ranges

* debugs indexing strategies

* Can select slot blocks

* finding number of pad cells

* Implements building slot tree

* finishes implementing slot builder

* Adds check that block size is a multiple of cell size

* Cleanup slotbuilder

* Review comments by Tomasz

* Fixes issue where ecK was used as numberOfSlots.

* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* remove new codecs for now

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* fix bad `elements` iter

* bump

* bump

* wip

* reworking slotbuilder

* move out of manifest

* expose getCidAndProof

* import index strat...

* remove getMHash

* remove unused artifacts

* alias zero

* add digest for multihash

* merge issues

* remove unused hashes

* add option to result converter

* misc

* fix tests

* add helper to derive EC block count

* rename method

* misc

* bump

* extract slot root building into own proc

* revert to manifest to accessor

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2024-01-08 14:52:46 -08:00
Dmitriy Ryajov b8ee2ac71e
Update multicodecs (#665)
* rework merkle tree support

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* style

* adding codex merkle and coders tests

* use default hash codec

* proof size changed

* add from nodes test

* shorte file names

* wip poseidon tree

* shorten file names

* root returns a result

* import poseidon tests

* update multicodecs

* consolidating codex types and adding new codecs

* update codec

* remove temp codecs constants

* move codecs related stuff out

* updating codecs

* misc

* updating sizes since block size was adjusted to 64kb

* fix merge issues and cleanup a few warnings
2023-12-22 13:04:01 +01:00
Dmitriy Ryajov 52c5578c46
Rework merkle tree (#654)
* rework merkle tree support

* deps

* rename merkletree -> codexmerkletree

* treed and proof encoding/decoding

* small change to invoke proof verification

* rename merkletree to codexmerkletree

* style

* adding codex merkle and coders tests

* fixup imports

* remove new codecs for now

* bump deps

* adding trace statement

* properly serde of manifest block codecs

* use default hash codec

* add more trace logging to aid debugging

* misc

* remove double import

* revert un-needded change

* proof size changed

* bump poseidon2

* add from nodes test

* shorte file names

* remove upraises

* wip poseidon tree

* adjust file names

* misc

* shorten file names

* fix bad `elements` iter

* don't do asserts

* add fromNodes and converters

* root and getProof now return result

* add poseidon2 tree tests

* root now returns result

* misc

* had to make merkletree a ref, because nim blows up otherwise

* root returns a result

* root returns a result

* import poseidon tests

* bump

* merkle poseidon2 digest

* misc

* add merkle digest tests

* bump

* don't use checksuite

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update codex/merkletree/merkletree.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* Update tests/codex/merkletree/generictreetests.nim

Co-authored-by: markspanbroek <mark@spanbroek.net>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

* missing return

* make toBool private (it's still needed otherwise comparison won't work)

* added `digestTree` that returns a tree and `digest` for root

* test against both poseidon trees - codex and poseidon2

* shorten merkle tree names

* don't compare trees - it's going to be too slow

* move comparison to mekrle helper

* remove merkle utils

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-21 06:41:43 +00:00
Eric f293082ae9
Reverts logging-proxy, commit 27f585eb6f (#660) 2023-12-20 13:24:40 +11:00
Eric 27f585eb6f
feat: create logging proxy (#653)
* implement a logging proxy

The logging proxy:
- prevents the need to import chronicles (as well as export except toJson),
- prevents the need to override `writeValue` or use or import nim-json-seralization elsewhere in the codebase, allowing for sole use of utils/json for de/serialization,
- and handles json formatting correctly in chronicles json sinks

* Rename logging -> logutils to avoid ambiguity with common names

* clean up

* add setProperty for JsonRecord, remove nim-json-serialization conflict

* Allow specifying textlines and json format separately

Not specifying a LogFormat will apply the formatting to both textlines and json sinks.

Specifying a LogFormat will apply the formatting to only that sink.

* remove unneeded usages of std/json

We only need to import utils/json instead of std/json

* move serialization from rest/json to utils/json so it can be shared

* fix NoColors ambiguity

Was causing unit tests to fail on Windows.

* Remove nre usage to fix Windows error

Windows was erroring with `could not load: pcre64.dll`. Instead of fixing that error, remove the pcre usage :)

* Add logutils module doc

* Shorten logutils.formatIt for `NBytes`

Both json and textlines formatIt were not needed, and could be combined into one formatIt

* remove debug integration test config

debug output and logformat of json for integration test logs

* Use ## module doc to support docgen
2023-12-19 22:12:47 +00:00
Eric df6b9c6760
Workaround for Hardhat last block timestamp bug (#644)
* Workaround for Hardhat timestamp bug

Likely due to a Hardhat bug in which the callbacks for subscription events are called and awaited before updating its local understanding of the last block time, Hardhat will report a block time in the `newHeads` event that is generally 1 second before the time reported from `getLatestBlock.timestamp`. This was causing issues with the OnChainClock's offset and therefore the `now()` used by the `OnChainClock` would sometimes be off by a second (or more), causing tests to fail.

This commit introduce a `codex_use_hardhat` compilation flag, that when set, will always get the latest block timestamp from Hardhat via the `getLatestBlock.timestamp` RPC call for `OnChainClock.now` calls. Otherwise, the last block timestamp reported in the `newHeads` event will be used.

Update the docker dist tests compilation flag for simulated proof failures (it was not correct), and explicitly add the `codex_use_hardhat=false` for clarity.

* enable simulated proof failures for coverage

* comment out failing test on linux -- will be replaced

* bump codex contracts eth

* add back clock offset for non-hardhat cases

* bump codex-contracts-eth

increases pointer by 67 blocks each period increase

* Add `codex_use_hardhat` flag to coverage tests
2023-12-20 08:06:24 +11:00
Eric 2c621e0421
chore: improve log information (#656)
* Add proof downtime to validator logs

* Add previous state to state machine transition log
2023-12-19 04:29:18 +00:00
Eric 39554c8ea9
sales was never marked as running and caused a hang (#655) 2023-12-19 01:28:17 +00:00
Ben Bierens f1b1fe9152
Adds endpoint to expose quota information (#652) 2023-12-14 10:57:16 +00:00
Adam Uhlíř e62cb96bde
feat: cleanup returning availability (#650)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-12-13 20:58:17 +01:00
Ben Bierens 0c3d1dd563
Verifiable manifests (#642)
* Adds test for encoding/decoding protected manifest

* Setting up verifiable manifest

* mysterious mysteries

* Successful encoding test for verifiable manifests

* extracts toF out of users of manifest code

* Update codex/manifest/coders.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>

* Review comments by Dmitriy

* Adds missing verifiable print to $ method.

* Replace poseidon2 F type with int as temporary stand-in for verification hashes

* Replaces verification hash placeholder with CID

---------

Signed-off-by: Ben Bierens <39762930+benbierens@users.noreply.github.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-12-12 08:11:54 +00:00
Ben Bierens a9f8090bb4
Add challenge to prove callback (#649)
* Sets up passing proof challenge to onProve callback

* Implements tests for proof challenge

* Removes unused import
2023-12-11 11:29:15 +01:00
Eric 3907ca4095
Add get active slot /slots/{slotId} to REST api, use utils/json (#645)
* Add get active slot /slots/{slotId} to REST api, use utils/json

- Add endpoint /slots/{slotId} to get an active SalesAgent from the Sales module. Used in integration tests to test when a sale has reached a certain state. Those integration test changes will be included in a larger PR, coming later.
- Add OpenAPI changes for new endpoint and associated components
- Use utils/json instead of nim-json-serialization. Required exemption of imports from several packages that export nim-json-serialization by default.

* Only except `toJson` from import/export of chronicles
2023-12-07 12:16:36 +11:00
Eric 86a6ef9215
remove waitFor cancelAndWait in proving state (#643)
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
2023-12-06 14:20:14 +11:00
Adam Uhlíř b38146d3f7
fix: check expiration is before request end (#641) 2023-12-05 14:25:28 +01:00
Adam Uhlíř 0e4387d1b3
refactor: move expiry update from fetchBatched (#634) 2023-11-28 22:04:11 +01:00
Dmitriy Ryajov 22c31046a7
Dont dial self (#633)
* don't dial self

* revert style changes

* filter out self on dial

* add helpers to `peerId` and `isSelf`

* don't fire up discovery eaguerly

* allow excluding multiple peers in sendWantHave

* revert style changes

* move self check to discovery.find

* readd eaguer dht lookup is required in some cases

* revert style changes

* misc

* drop peer first, before queueing a dht lookup

* moar style changes

* use isSelf
2023-11-27 10:25:53 -08:00
markspanbroek b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Adam Uhlíř 79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00
Adam Uhlíř 8681a40ee7
feat: update expiry when data downloaded and slot filled (#619)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 10:09:12 +00:00
Dmitriy Ryajov ec8d0c98b2
Fix REST endpoints semantics (#612)
* Fix REST endpoints semantics

* update endpoint description

* update, operation id

* Adding enum support

* make enum descerializer public

* add support for listing manifests

* test `/data` endpoint to list local manifests

* debug leftovers

* remove commented out line
2023-11-20 16:14:06 -08:00
Tomasz Bekas 70efd13c73
Fix block delete by treeCid and index (#623) 2023-11-20 14:48:56 +01:00
Ben Bierens bece1b88a1
Feat/bump questionable (#627)
* Bumps questionable version to 0.10.12

* removes unnecessary questionnable bindings.

* Fixes tests

* unnecessary whitespaces
2023-11-17 13:49:45 +01:00
Ben Bierens 40d7714a67
Fixes json formatting for NodeId (#614) 2023-11-15 13:29:17 +00:00
Dmitriy Ryajov 4d50ecf504
handle exceptions gracefuly in rest api (#617) 2023-11-14 11:05:43 -08:00
Dmitriy Ryajov 8d61391073
Light cleanup (#616)
mostly whitespaces cleanup
2023-11-14 09:52:27 -08:00
Dmitriy Ryajov a1725594a8
Fix json serialization for cid and enums (#615)
* use str on JString types, `$` will preserve `"`

* Adding enum support

* deserialize cid test

* make enum descerializer public

* unify fromJson for objects and refs

* add enum descerialization testing
2023-11-14 07:23:50 -08:00
Tomasz Bekas 2396c4d76d
Blockexchange uses merkle root and index to fetch blocks (#566)
* Blockexchange uses merkle root and index to fetch blocks

* Links the network store getTree to the local store.

* Update codex/stores/repostore.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>

* Rework erasure.nim to include recent cleanup

* Revert accidential changes to lib versions

* Addressing review comments

* Storing proofs instead of trees

* Fix a comment

* Fix broken tests

* Fix for broken testerasure.nim

* Addressing PR comments

---------

Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-14 13:02:17 +01:00
Ben Bierens 34a9c5e88d
Storage/purchases endpoint (#609)
* adds list of purchase ids to storage API

* adds list of purchase ids to storage API

* Updates openAPI

* fixes openAPI type
2023-11-13 11:30:27 +00:00
Ben Bierens 80e2ef4eb5
Debug/info shows announce addresses (#610)
* Shows announce addresses in debug/info endpoint.

* Shows announce addresses in debug/info endpoint.
2023-11-12 09:58:40 +01:00
Ben Bierens cb02962231
Adds endpoint for listing files (manifests) in node. Useful for demo UI. (#599)
* Adds endpoint for listing files (manifests) in node. Useful for demo UI.

* Moves upload/download/files into content API calls.

* Cleans up json serialization for manifest

* Cleans up some more json serialization

* Moves block iteration and decoding to node.nim

* Moves api methods into their own init procs.

* Applies RestContent api object.

* Replaces format methods with Rest objects in json.nim

* Unused import

* Review comments by Adam

* Fixes issue where content/local endpoint clashes with content/cid.

* faulty merge resolution

* Renames content API to data.

* Fixes faulty rebase

* Adds test for data/local API

* Renames local and download api.
2023-11-09 08:47:09 +00:00
Eric 7d4ea878d2
Support logging to file (#558)
* Support logging to file

* Log the entire config and fix build error

* Downgrade log level for "starting codex node" config output

* bump ethers to prevent nonce gaps

* fix tests
2023-11-09 16:35:55 +11:00
Adam Uhlíř c0bec2f899
feat: ensure block expiry (#597)
* feat: update block expiry

* chore: feedback implementation

* chore: feedback implementation

* chore: feedback implementation
2023-11-06 08:10:30 +00:00
Slava 11cd2c46ad
Rename metrics (#478) (#603) 2023-11-03 17:21:54 +02:00
Adam Uhlíř 2fc71cf81b
feat: partial payouts for cancelled requests (#561) 2023-10-24 10:12:54 +00:00
Dmitriy Ryajov 8a7d74e6b2
removing old por proofs implementation (#593) 2023-10-23 07:58:07 -07:00
Adam Uhlíř c28627d16f
chore(logging): purchases logging (#567) 2023-10-19 10:12:49 +02:00
Adam Uhlíř 5f4218ba1c
fix: use increaseAllowance for token approving (#565) 2023-10-16 13:09:54 +02:00
Ben Bierens 040434aa19
renames log properties named 'count' (#560) 2023-10-03 13:33:01 +02:00
Ben Bierens fede106118
Numbered log entries (#559)
* add numbered log entries (hack)

* Changed from CLI argument to compiler define

* Use uint64 for counting

---------

Co-authored-by: gmega <giuliano.mega@gmail.com>
2023-10-03 11:53:58 +02:00
Eric 570a1f7b67
[marketplace] Availability improvements (#535)
## Problem
When Availabilities are created, the amount of bytes in the Availability are reserved in the repo, so those bytes on disk cannot be written to otherwise. When a request for storage is received by a node, if a previously created Availability is matched, an attempt will be made to fill a slot in the request (more accurately, the request's slots are added to the SlotQueue, and eventually those slots will be processed). During download, bytes that were reserved for the Availability were released (as they were written to disk). To prevent more bytes from being released than were reserved in the Availability, the Availability was marked as used during the download, so that no other requests would match the Availability, and therefore no new downloads (and byte releases) would begin. The unfortunate downside to this, is that the number of Availabilities a node has determines the download concurrency capacity. If, for example, a node creates a single Availability that covers all available disk space the operator is willing to use, that single Availability would mean that only one download could occur at a time, meaning the node could potentially miss out on storage opportunities.

## Solution
To alleviate the concurrency issue, each time a slot is processed, a Reservation is created, which takes size (aka reserved bytes) away from the Availability and stores them in the Reservation object. This can be done as many times as needed as long as there are enough bytes remaining in the Availability. Therefore, concurrent downloads are no longer limited by the number of Availabilities. Instead, they would more likely be limited to the SlotQueue's `maxWorkers`.

From a database design perspective, an Availability has zero or more Reservations.

Reservations are persisted in the RepoStore's metadata, along with Availabilities. The metadata store key path for Reservations is ` meta / sales / reservations / <availabilityId> / <reservationId>`, while Availabilities are stored one level up, eg `meta / sales / reservations / <availabilityId> `, allowing all Reservations for an Availability to be queried (this is not currently needed, but may be useful when work to restore Availability size is implemented, more on this later).

### Lifecycle
When a reservation is created, its size is deducted from the Availability, and when a reservation is deleted, any remaining size (bytes not written to disk) is returned to the Availability. If the request finishes, is cancelled (expired), or an error occurs, the Reservation is deleted (and any undownloaded bytes returned to the Availability). In addition, when the Sales module starts, any Reservations that are not actively being used in a filled slot, are deleted.

Having a Reservation persisted until after a storage request is completed, will allow for the originally set Availability size to be reclaimed once a request contract has been completed. This is a feature that is yet to be implemented, however the work in this PR is a step in the direction towards enabling this.

### Unknowns
Reservation size is determined by the `StorageAsk.slotSize`. If during download, more bytes than `slotSize` are attempted to be downloaded than this, then the Reservation update will fail, and the state machine will move to a `SaleErrored` state, deleting the Reservation. This will likely prevent the slot from being filled.

### Notes
Based on #514
2023-09-29 14:33:08 +10:00
Adam Uhlíř 2f1c778d02
fix: unknown state goes to payout when slot state is finished (#555) 2023-09-27 15:57:41 +02:00
Dmitriy Ryajov 25ea7fd0b2
Erasure cleanup (#545)
* cleanup erasure coding

* moar cleanup

* fix off by 1 issues in tests

* style

* consolidate decoding data code

* simplify tuple unpacking

* fix retrieve purchase

We don't support single blocks for now

* Apply suggestions from code review

Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>

---------

Signed-off-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2023-09-25 07:31:10 -07:00
markspanbroek 71cd35112b
Taiko L2 (#483)
* [docs] fix two client scenario: add missing collateral

* [integration] separate step to wait for node to be started

* [cli] add option to specify ethereum private key

* Remove unused imports

* Fix warnings

* [integration] move type definitions to correct place

* [integration] wait a bit longer for a node to start in debug mode

When e.g. running against Taiko test net rpc, the node start
takes longer

* [integration] simplify handling of codex node and client

* [integration] add Taiko integration test

* [contracts] await token approval confirmation before next tx

* [contracts] deployment address of marketplace on Taiko

* [cli] --eth-private-key now takes a file name

Instead of supplying the private key on the command line,
expect the private key to be in a file with the correct
permissions.

* [utils] Fixes undeclared `activeChroniclesStream` on Windows

* [build] update nim-ethers to include PR #52

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [cli] Better error messages when reading eth private key

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [integration] simplify reading of cmd line arguments

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [build] update to latest version of nim-ethers

* [contracts] updated contract address for Taiko L2

* [build] update codex contracts to latest version

---------

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2023-09-13 16:17:56 +02:00
Adam Uhlíř ae89db1eea
fix: sales concurrency bug (#537) 2023-09-05 16:47:29 +02:00
markspanbroek d3a22a7b7b
Fix slot queue push (#542)
* [sales] remove availability check before adding to slot queue

* [sales] add missing return statement

* [tests] remove 'eventuallyCheck' helper

* [sales] remove reservations from slot queue

* [tests] rename module `eventually` -> `always`

* [sales] increase slot queue size

Because it will now also hold items for which we haven't
checked availability yet.
2023-09-04 16:42:09 +02:00
Ben Bierens d279eebd69
Start discovery after announce address is updated (#536)
* Start discovery after announce address is updated

* Moves switch start-stop to codex.nim. Adjusts timing to solve issue
2023-09-04 11:12:20 +02:00
Ben Bierens 545e0d47e1
Slows down block maintenance iteration (#543)
* Slows down block maintenance iteration

* Fixes unstable integration test.
2023-09-04 11:12:14 +02:00
Eric 37b3d99c3d
Improve integration testing client (CodexClient) and json serialization (#514)
* Improve integration testing client (CodexClient) and json serialization

The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly.

In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested.

These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier.

* Update integration test to use nodes=2 and tolerance=1

* clean up
2023-09-01 15:44:41 +10:00
Adam Uhlíř 098baf47d1
chore: improve logging (#526) 2023-08-28 11:33:44 +02:00
Ben Bierens f10c3f3934
Debug/extra logging (#509)
* logs

* poll log

* more logs

* time log for upload steps

* Adds metric and log for block retrieval time

* adds some logging

* move logging for storestream block indices

* Log at start and end of block iteration

* applies branch blockstore-bugfix

* Cleanup

* Cleanup
2023-08-22 08:35:16 +02:00
Ben Bierens 238771eb73
Repostore crash debug (#521)
* repostore listBlocks reads too much data

* Add assert to make sure data is not queried
2023-08-22 08:26:58 +02:00
Adam Uhlíř f459a2c6f6
refactor: merging proving module into sales (#469)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2023-08-21 12:26:43 +02:00
Eric 9cecb68520
[repostore] Retrieve empty blocks (#513)
Add handling of empty blocks in the RepoStore.

* Add empty block handling to repostore for put, del, has
Also added tests for all empty block handling blockstore operations. This showed there was an ambiguous identifier present for `hasBlock`, so one of the two `hasBlock` definitions was removed in `repostore`.

* Change CacheStore to RepoStore in testerasure
As CacheStore is not used in the node, update the Datastore used in the erasure coding tests to be a RepoStore. This ensures that the K > 1 cases are being tested, where they will produce empty padding blocks in the erasure-coded manifests.
2023-08-21 12:51:04 +10:00
Tomasz Bekas e8601274b9
Merkle tree construction (#504)
* Building a merkle tree

* Obtaining merkle proof from a tree

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-08-15 13:23:35 +02:00
Adam Uhlíř 39efac1a97
fix: load slots on sales module start (#510) 2023-08-15 11:39:49 +02:00
Jaremy Creechley 7efa9177df
Bump deps take2 (#492)
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-08-01 16:47:57 -07:00
Eric 3e80de3454
Cancel and wait for asyncstatemachine futures when stopping (#493)
* Simplify `.then` (promise api) and tests

* Remove tracked future when cancelled. Add tracked future tests

* Track and cancel statemachine futures

The futures created in each asyncstatemachine instance are tracked, and each future is cancelled and waited in `stop`.

Change `asyncstatemachine.stop` to be async so `machine.trackedFutures.cancelAndWait` could be called.
Add a constructor for `asyncstatemachine` that initialises the `trackedFutures` instance, and call the constructor from derived class constructors.
2023-07-31 15:09:34 +10:00
Eric 1d161d383e
Slot queue (#455)
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).

Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.

When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.

Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used

The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).

When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.

SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.

Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.

Prevent loading request from contract (network request) if there is an existing item in queue for that request.

Check availability before adding request to queue.

Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).

#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop

Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).

Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
  - Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks

#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.

### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
  - prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating

An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.

Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.

Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
  - remove unneeded error handling as pragmas were moved

Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.

Sales tests: shut down repo after sales stop, to fix SIGABRT in CI

### Add async Promise API
  - modelled after JavaScript Promise API
  - alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
  - Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
  - Add cancellation callbacks to utils/then, ensuring cancellations are handled properly

## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb

---------

Co-authored-by: Jaremy Creechley <creechley@gmail.com>
2023-07-25 12:50:30 +10:00
Ben Bierens 14c5270e83
Add metrics (#478)
* Adds metrics to block exchange.

* Adds metrics to purchasing

* Adds metrics to upload and download API

* Adds metrics to the repo store

* Fixes exception in repostore start.

* Merge managed to mess up indentation.
2023-07-20 09:56:28 +02:00
Ben Bierens 6708202a5f
Makes rootHash private to Manifest object. (#488)
* Makes rootHash private to Manifest object.

* Locks down all fields of Manifest.

* Review comments by Mark
2023-07-19 16:06:59 +02:00
Tomasz Bekas 3bb5960739
Fix for decoding large manifests (#479) 2023-07-19 15:10:14 +02:00
Ben Bierens 7227a4a38d
Restores the three key lines that make peer discovery work. (#484) 2023-07-18 07:51:20 +02:00
jessiebroke ac2aca42b1
Block exchange debug (#426)
* Adds sequence diagram for local download

* Adds network download flow

* makes node properly request block when presence-list is received

* starting to work :O

* Fixes fully connected download test up to 20 nodes.

* Better sequence diagram of network block resolution

* Cleans up comments and traces.

* Accidentally deleted important else

* Review comments by Dmitriy

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-07-18 07:50:47 +02:00
markspanbroek 3879ec8e3a
[clock] waitUntil() completes immediately when block arrives (#475)
Previously it could take up to one second to complete
the future. This messed with the timings in the
integration tests and made them less predictable.
2023-07-13 11:19:45 +02:00
Jaremy Creechley 711e5e09d1
Plumbing in conf types (#472)
Goal is to provide documentation and to enable conf utils byte params.

* Create byte units NByte and plumb through in size locations.
* add json serde
* plumb parseDuration to conf
2023-07-06 16:23:27 -07:00
Dmitriy Ryajov bd594c9aaf
Create memory-leak detecting test suite (#226)
* adding tracker for streamstore

* adding tracker tests

* Sets up tracker helper functions and closes streams in testnode.nim

* Deploying checksuite for memory leak tracker checking.

* Successfully deploys checksuite and asyncchecksuite.

* Fix leak in testpor.nim

* Fixes leaked storestream in testnetwork.nim

* Fixes integration tests

* Cleanup

* cleanup comment by Mark

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-06-22 12:01:21 -06:00
Jaremy Creechley e47b38af11
Improving proc/func formatting consistency (#454)
* Fixes/workarounds for nimsuggest failures in codex.nim.

* remove rng prefix - it appears to work now

* format new's to be more consistent

* making proc formatting a bit more consistent
2023-06-22 08:11:18 -07:00
Eric Mastro 2a92dc9702
[marketplace] Simulate invalid proof submissions (#393) 2023-06-22 20:32:18 +10:00
Jaremy Creechley 6d1469b4be
Nimsuggest tweaks (#453)
* Fixes/workarounds for nimsuggest failures in codex.nim.
* remove rng prefix - it appears to work now
2023-06-21 15:02:05 -07:00
Adam Uhlíř cfd2cf9302
feat: bigint uses decimal over hex encoding (#452) 2023-06-21 07:46:18 +02:00
Adam Uhlíř 4cd8dd2e05
feat: slots rest api (#443)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-06-20 14:52:15 +02:00
Adam Uhlíř 66b5f23b65
chore: remove unused imports (#445) 2023-06-19 11:53:41 +02:00
Ben Bierens db9d90b465
Dht debug (#446)
* Support for building docker images with local modifications for the purpose of testing and debugging

* exposes peer information via debug/info

* api-names slightly kinder to json serializers

* Moves debug image-building

* fixes misalignment of debug peer info array

* Changes switchPeers source from KeyBook to AddressBook (filed ticket in libp2p, discussed with Tanguy)

* Limited success with dist-test peer discovery tests

* Removes unnecessary random-timer

* bumps dht repo. Adds peerId to formatNode

* Removes unused prints

* bumps libp2p-dht

* Exposes node address on debug api

* Adds traces

* review comments by me

* Hides debug/peers api behind compile flag

* Waiting for nim-libp2p-dht PR merge

* bumps libp2p-dht back to main after PRs were merged there.

* Cleanup
2023-06-19 08:21:03 +02:00
Adam Uhlíř 4f99d88c0a
feat: allow any case for logLevel (#444) 2023-06-16 18:20:49 +02:00
Adam Uhlíř 720c372be0
fix(api): set content type for json (#441) 2023-06-13 15:33:01 +02:00
markspanbroek 3181361658
Cleanup purchasing state machine (#422)
* [state machine] Allow querying of state properties

* [purchasing] use new state machine

* [state machine] remove old state machine implementation

* [purchasing] remove duplication in error handling
2023-06-05 10:48:06 +02:00
Adam Uhlíř c075890737
feat: storage host collects reward (#416)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-05-15 09:02:57 +02:00
Adam Uhlíř 4d028c6cb3
feat: contract address management (#405)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-05-03 09:24:25 +02:00
Adam Uhlíř ff6cc01857
feat: adding config and env. variable support (#379)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-05-02 15:06:34 +02:00
markspanbroek 515304f9c5
[node] fix: stop validator (#401) 2023-05-01 16:23:26 +02:00
markspanbroek d56eb6aee1
Validator (#387)
* [contracts] Add SlotFreed event

* [integration] allow test node to be stopped twice

* [cli] add --validator option

* [contracts] remove dead code

* [contracts] instantiate OnChainMarket and OnChainClock only once

* [contracts] add Validation

* [sales] remove duplicate import

* [market] add missing import

* [market] subscribe to all SlotFilled events

* [market] add freeSlot()

* [sales] fix warnings

* [market] subscribe to SlotFreed events

* [contracts] fix warning

* [validator] keep track of filled slots

* [validation] remove slots that have ended

* [proving] absorb Proofs into Market

Both Proofs and Market are abstractions around
the Marketplace contract, having them separately
is more trouble than it's worth at the moment.

* [market] add markProofAsMissing()

* [clock] speed up waiting for clock in tests

* [validator] mark proofs as missing

* [timer] fix error on node shutdown

* [cli] handle --persistence and --validator separately

* [market] allow retrieval of proof timeout value

* [validator] do not subscribe to SlotFreed events

Freed slots are already handled in
removeSlotsThatHaveEnded(), and onSlotsFreed()
interfered with its iterator.

* [validator] Start validation at the start of a new period

To decrease the likelihood that we hit the validation timeout.

* [validator] do not mark proofs as missing after timeout

* [market] check whether proof can be marked as missing

* [validator] simplify validation

Simulate a transaction to mark proof as missing, instead
of trying to keep track of all the conditions that may
lead to a proof being marked as missing.

* [build] use nim-ethers PR #40

Uses "pending" blocktag instead of "latest" blocktag
for better simulation of transactions before sending
them.

https://github.com/status-im/nim-ethers/pull/40

* [integration] integration test for validator

* [validator] monitor a maximum number of slots

Adds cli parameter --validator-max-slots.

* [market] fix missing collateral argument

After rebasing, add the new argument to fillSlot calls.

* [build] update to nim-ethers 0.2.5

* [validator] use Set instead of Table to keep track of slots

* [validator] add logging

* [validator] add test for slot failure

* [market] use "pending" blocktag to use more up to date block time

* [contracts] remove unused import

* [validator] fix: wait until after period ends

The smart contract checks that 'end < block.timestamp',
so we need to wait until the block timestamp is greater
than the period end.
2023-04-19 15:06:00 +02:00
Adam Uhlíř 131d003a0c
feat: collateral per slot (#390)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-04-14 11:04:17 +02:00
Dmitriy Ryajov 86a3f74448
adding some clarification comments (#391) 2023-04-10 14:45:10 -06:00
Eric Mastro ccf349bd14
[marketplace] Add Reservations Module (#340)
* [marketplace] reservations module

- add de/serialization for Availability
- add markUsed/markUnused in persisted availability
- add query for unused
- add reserve/release
- reservation module tests
- split ContractInteractions into client contracts and host contracts
- remove reservations start/stop as the repo start/stop is being managed by the node
- remove dedicated reservations metadata store and use the metadata store from the repo instead
- Split ContractInteractions into:
  - ClientInteractions (with purchasing)
  - HostInteractions (with sales and proving)
- compilation fix for nim 1.2

[repostore] fix started flag, add tests

[marketplace] persist slot index
For loading the sales state from chain, the slot index was not previously persisted in the contract. Will retrieve the slot index from the contract when the sales state is loaded.

* Revert repostore changes

In favour of separate PR https://github.com/status-im/nim-codex/pull/374.

* remove warnings

* clean up

* tests: stop repostore during teardown

* change constructor type identifier

Change Contracts constructor to accept Contracts type instead of ContractInteractions.

* change constructor return type to Result instead of Option

* fix and split interactions tests

* clean up, fix tests

* find availability by slot id

* remove duplication in host/client interactions

* add test for finding availability by slotId

* log instead of raiseAssert when failed to mark availability as unused

* move to SaleErrored state instead of raiseAssert

* remove unneeded reverse

It appears that order is not preserved in the repostore, so reversing does not have the intended effect here.

* update open api spec for potential rest endpoint errors

* move functions about available bytes to repostore

* WIP: reserve and release availabilities as needed

WIP: not tested yet

Availabilities are marked as used when matched (just before downloading starts) so that future matching logic does not match an availability currently in use.

As the download progresses, batches of blocks are written to disk, and the equivalent bytes are released from the reservation module. The size of the availability is reduced as well.

During a reserve or release operation, availability updates occur after the repo is manipulated. If the availability update operation fails, the reserve or release is rolled back to maintain correct accounting of bytes.

Finally, once download completes, or if an error occurs, the availability is marked as unused so future matching can occur.

* delete availability when all bytes released

* fix tests + cleanup

* remove availability from SalesContext callbacks

Availability is no longer used past the SaleDownloading state in the state machine. Cleanup of Availability (marking unused) is handled directly in the SaleDownloading state, and no longer in SaleErrored or SaleFinished. Likewise, Availabilities shouldn’t need to be handled on node restart.

Additionally, Availability was being passed in SalesContext callbacks, and now that Availability is only used temporarily in the SaleDownloading state, Availability is contextually irrelevant to the callbacks, except in OnStore possibly, though it was not being consumed.

* test clean up

* - remove availability from callbacks and constructors from previous commit that needed to be removed (oopsie)
- fix integration test that checks availabilities
  - there was a bug fixed that crashed the node due to a missing `return success` in onStore
  - the test was fixed by ensuring that availabilities are remaining on the node, and the size has been reduced
- change Availability back to non-ref object and constructor back to init
- add trace logging of all state transitions in state machine
- add generally useful trace logging

* fixes after rebase

1. Fix onProve callbacks
2. Use Slot type instead of tuple for retrieving active slot.
3. Bump codex-contracts-eth that exposes getActivceSlot call.

* swap contracts branch to not support slot collateral

Slot collateral changes in the contracts require further changes in the client code, so we’ll skip those changes for now and add in a separate commit.

* modify Interactions and Deployment constructors

- `HostInteractions` and `ClientInteractions` constructors were simplified to take a contract address and no overloads
- `Interactions` prepared simplified so there are no overloads
- `Deployment` constructor updated so that it takes an optional string parameter, instead `Option[string]`

* Move `batchProc` declaration

`batchProc` needs to be consumed by both `node` and `salescontext`, and they can’t reference each other as it creates a circular dependency.

* [reservations] rename `available` to `hasAvailable`

* [reservations] default error message to inner error msg

* add SaleIngored state

When a storage request is handled but the request does match availabilities, the sales agent machine is sent to the SaleIgnored state. In addition, the agent is constructed in a way that if the request is ignored, the sales agent is removed from the list of active agents being tracked in the sales module.
2023-04-04 17:05:16 +10:00
Adam Uhlíř 3198db414d
fix: approving tokens transfer when creating request (#385) 2023-03-30 11:34:38 +02:00
markspanbroek 4ffe7b8e06
Generate proofs when required (#383)
* [maintenance] speedup integration test

* [rest api] add proofProbability parameter to storage requests

* [integration] negotiation test ends when contract starts

* [integration] reusable 2 node setup for tests

* [integration] introduce CodexClient for tests

* [node] submit storage proofs when required

* [contracts] Add Slot type

* [proving] replace onProofRequired & submitProof with onProve

Removes duplication between Sales.onProve() and
Proving.onProofRequired()
2023-03-27 15:47:25 +02:00
Eric Mastro c9a62de13f
fix: repostore started flag in stop() (#374) 2023-03-16 09:00:36 -06:00
Adam Uhlíř 5ae545bc3c
docs: openapi specification (#355)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-03-15 14:10:53 +01:00
c-blake 7cdd4f79e3
Resuscitate https://github.com/status-im/nim-codex/pull/251 except for (#358)
A) renaming to --repo-kind which makes more sense in the current code
and B) without a working cacheStore (yet)

Tagging related https://github.com/status-im/nim-codex/issues/357
2023-03-14 16:32:15 -06:00
Adam Uhlíř 6cbf3ab044
fix(api): correctly using tolerance for storage request (#359) 2023-03-13 13:23:50 +01:00
Ben Bierens da79660f8e
Enable stylecheck (#353)
* applying styleCheck

* stuck on vendor folder

* Applies style check

* Turns styleCheck back off

* switches to stylecheck:usages

* Fixes empty template casing

* rolls up nim-blscurve, nim-datastore, nim-ethers, nim-leopard, and nim-taskpools.

* bumps nim-confutils and removes unused import from fileutils.nim

* Unused using in fileutils.nim is required by CI

* Reverts bump of nim-confutils module
2023-03-10 08:02:54 +01:00
markspanbroek 7a0a48e4a5
Fix warnings (drops Nim 1.2) (#348)
* [build] disable XCannotRaiseY hint

There are too many {.raises:[Defect].} in the
libraries that we use, drowning out all other
warnings and hints

* [build] disable BareExcept warning

Not yet enabled in a released version of Nim,
so libraries that we depend on have not fixed
this yet, drowning out our own hints and warnings

* [build] disable DotLikeOps warning

dot-like ops were an experiment that is not going
land in Nim

* [build] compile log statements in tests

When running tests, all log statements are compiled.
They are filtered out at runtime during a test run.

* [build] do not build executable when running unit test

It's already built in the integration test

* [build] Fix warnings

- remove unused code
- remove unused imports
- stop using deprecated stuff

* [build] Put compiler flags behind nim version checks

* [CI] remove Nim 1.2 compatibility
2023-03-09 12:23:45 +01:00
Ben Bierens 9c8a59d150
Blockstore maintenance (#347)
* setting up

* Implements timer utility

* applies async and cancellation to timer loop

* Sets up mocktimer and mockblockstore to set up first maintenance module test

* wip: first test that calls blockChecker

* wip: adding user type to timer callback

* Chronicles doesn't like type-arguments? Disables logging in timer module for now

* Implementing block check test for blockMaintainer module

* Sets up additional tests for blockmaintainer

* Removes generic from timer module. Implements numberOfBlocks per interval in blockMaintainer.

* Implements blockMaintainer

* Sets up tests for blockChecker

* Some comments by Mark

* Cleanup repostore tests

* Setting up the use of std/times for block TTL tracking

* repostore adds expiration timestamp

* Defaults the repostore clock to the system clock

* Applies updates to repostore interface.

* Implements retrieving of block expiration information from repostore

* Sets up tests for maintenance module behavior

* Implements block maintenance module

* Wires maintenance module into codex. Sets up integration tests for block expiration

* Sets up test for missing behavior: removing timestamp metadata on block delete

* Implements removing of expiration metadata in repoStore

* Fixes integration tests for block expiration

* Adds block expiration tests to integration test run

* Handled some comments by Dmitriy

* Review comment by Dmitriy: Removes seq[cid] from runBlockCheck

* Review comment by Dmitriy: Moves key formatting methods to keyutils.

* Review comment by Dmitriy: Encodes durations using chronos

* Fixes conversion of TTL type in conf.

* Review comments by Dmitriy

* Adds unit tests for keyUtils.

* Adds test coverage for exception in maintenance module
2023-03-08 16:04:54 +01:00
Eric Mastro 25f68c1e4c
[marketplace] Load sales state from chain (#306)
* [marketplace] get active slots from chain

# Conflicts:
#	codex/contracts/market.nim

* [marketplace] make on chain event callbacks async

# Conflicts:
#	tests/codex/helpers/mockmarket.nim

* [marketplace] make availability optional for node restart

# Conflicts:
#	tests/codex/testsales.nim

* [marketplace] add async state machine

Allows for `enterAsync` to be cancelled.

* [marketplace] move sale process to async state machine

* [marketplace] sales state machine tests

* bump dagger-contracts

* [marketplace] fix ci issue with chronicles output

* PR comments

- add slotIndex to `SalesAgent` constructor
- remove `SalesAgent.init`
- rename `SalesAgent.init` to `start` and `SalesAgent.deinit` to `stop`.
- rename `SalesAgent. populateRequest` to `SalesAgent.retreiveRequest`.
- move availability removal to the downloading state. once availability is persisted to disk, it should survive node restarts.
-

* [marketplace] handle slot filled by other host

Handle the case when in the downloading, proving, or filling states, that another host fills the slot.

* [marketplace] use requestId for mySlots

* [marketplace] infer slot index from slotid

prevents reassigning a random slot index when restoring state from chain

* [marketplace] update to work with latest contracts

* [marketplace] clean up

* [marketplace] align with contract changes

- getState / state > requestState
- getSlot > getRequestFromSlotId
- support MarketplaceConfig
- support slotState, remove unneeded Slot type
- collateral > config.collateral.initialAmount
- remove proofPeriod contract call
- Revert reason “Slot empty” > “Slot is free”
- getProofEnd > read SlotState

Tests for changes

* [marketplace] add missing file

* [marketplace] bump codex-contracts-eth

* [config] remove unused imports

* [sales] cleanup

* [sales] fix: do not crash when fetching state fails

* [sales] make slotIndex non-optional

* Rebase and update NBS commit

Rebase on top of main and update NBS commit to the CI fix.

* [marketplace] use async subscription event handlers

* [marketplace] support slotIndex no longer optional

Previously, SalesAgent.slotIndex had been moved to not optional. However, there were still many places where optionality was assumed. This commit removes those assumuptions.

* [marketplace] sales state machine: use slotState

Use `slotState` instead of `requestState` for sales state machine.

* [marketplace] clean up

* [statemachine] adds a statemachine for async workflows

Allows events to be scheduled synchronously.

See https://github.com/status-im/nim-codex/pull/344

Co-Authored-By: Ben Bierens <thatbenbierens@gmail.com>
Co-Authored-By: Eric Mastro <eric.mastro@gmail.com>

* [market] make market callbacks synchronous

* [statemachine] export Event

* [statemachine] ensure that no errors are raised

* [statemachine] add machine parameter to run method

* [statemachine] initialize queue on start

* [statemachine] check futures before cancelling them

* [sales] use new async state machine

- states use new run() method and event mechanism
- StartState starts subscriptions and loads request

* [statemachine] fix unsusbscribe before subscribe

* [sales] replace old state transition tests

* [sales] separate state machine from sales data

* [sales] remove reference from SalesData to Sales

* [sales] separate sales context from sales

* [sales] move decoupled types into their own modules

* [sales] move retrieveRequest to SalesData

* [sales] move subscription logic into SalesAgent

* [sales] unsubscribe when finished or errored

* [build] revert back to released version of nim-ethers

* [sales] remove SaleStart state

* [sales] add missing base method

* [sales] move asyncSpawn helper to utils

* [sales] fix imports

* [sales] remove unused variables

* [sales statemachine] add async state machine error handling (#349)

* [statemachine] add error handling to asyncstatemachine

- add error handling to catch errors during state.run
- Sales: add ErrorState to identify which state to transition to during an error. This had to be added to SalesAgent constructor due to circular dependency issues, otherwise it would have been added directly to SalesAgent.
- Sales: when an error during run is encountered, the SaleErrorState is constructed with the error, and by default (base impl) will return the error state, so the machine can transition to it. This can be overridden by individual states if needed.

* [sales] rename onSaleFailed to onSaleErrored

Because there is already a state named SaleFailed which is meant to react to an onchain RequestFailed event and also because this callback is called from SaleErrored, renaming to onSaleErrored prevents ambiguity and confusion as to what has happened at the callback callsite.

* [statemachine] forward error to state directly

without going through a machine method first

* [statemachine] remove unnecessary error handling

AsyncQueueFullError is already handled in schedule()

* [statemachine] test that cancellation ignores onError

* [sales] simplify error handling in states

Rely on the state machine error handling instead
of catching errors in the state run method

---------

Co-authored-by: Mark Spanbroek <mark@spanbroek.net>

* [statemachine] prevent memory leaks

prevent memory leaks and nil access defects by:
- allowing multiple subscribe/unsubscribes of salesagent
- disallowing individual salesagent subscription calls to be made externally (requires the .subscribed check)
- allowing mutiple start/stops of asyncstatemachine
- disregard asyncstatemachine schedules if machine not yet started

* [salesagent] add salesagent-specific tests

1. test multiple subscribe/unsubscribes
2. test scheduling machine without being started
3. test subscriptions are working correctly with external events
4. test errors can be overridden at the state level for ErrorHandlingStates.

---------

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Mark Spanbroek <mark@spanbroek.net>
Co-authored-by: Ben Bierens <thatbenbierens@gmail.com>
2023-03-08 14:34:26 +01:00
Ben Bierens d263ca0735
Docker build (#354)
* Sets up working dockerized build and codex docker image creation

* Making codex configurable from the docker environment

* Sets up two networks with three codex nodes

* enables and exposes metrics endpoint for first node

* Manually performed two-client test scenario with docker containers

* Sets up docker-ignore and docker github workflow

* Wires up all codex CLI arguments to docker env vars

* Makes API_PORT variable optional as well

* Removes duplicate docker-login step

* Fixes path to docker file

* Switches target dockerhub for debugging

* Adds git tag info to --version output

* Exposes version information via debug endpoint

* Debugging docker image

* specifies target platforms for docker build

* specifies platform for QEMU and buildx steps

* Attempt to debug line endings

* Disables march-native in config.nims as test

* Applies make argument to disable architecture optimization during docker build

* Removes subset version tags from docker build

* Restore multi-arch build

* Removes docker-build test branch from CI branches
2023-03-08 12:45:55 +01:00
Dmitriy Ryajov 153d72c6c4
updating readme with latest changes (#331) 2023-02-15 13:40:21 -06:00
Eric Mastro df729be261
[marketplace] support latest contracts changes (#327)
* [marketplace] support removal of Storage contract

* [marketplace] change submod dep dagger-contracts to codex-contracts-eth
2023-01-19 16:58:04 +11:00
Dmitriy Ryajov 0beeefd760
Repo limits (#319)
* initial implementation of repo store

* allow isManifest on multicodec

* rework with new blockstore

* add raw codec

* rework listBlocks

* remove fsstore

* reworking with repostore

* bump datastore

* fix listBlocks iterator

* adding store's common tests

* run common store tests

* remove fsstore backend tests

* bump datastore

* add `listBlocks` tests

* listBlocks filter based on block type

* disabling tests in need of rewriting

* allow passing block type

* move BlockNotFoundError definition

* fix tests

* increase default advertise loop sleep to 10 mins

* use `self`

* add cache quota functionality

* pass meta store and start repo

* add `CacheQuotaNamespace`

* pass meta store

* bump datastore to latest master

* don't use os `/` as key separator

* Added quota limits support

* tests for quota limits

* add block expiration key

* remove unnesesary space

* use idleAsync in listBlocks

* proper test name

* re-add contrlC try/except

* add storage quota and block ttl config options

* clarify comments

* change expires key format

* check for block presence before storing

* bump datastore

* use dht with fixed datastore `has`

* bump datastore to latest master

* bump dht to latest master
2022-12-02 18:00:55 -06:00
markspanbroek cdd175279c
Fix nimble requirements (#305)
* [build] Update nim-ethers to 0.2.4

* [build] set correct lrucache requirement in nimble file
2022-11-16 13:10:23 +01:00
Dmitriy Ryajov 88f1be388a
fix `help`, `info`, add description for `loglevel` (#316)
* fix `help`, `info`, add description for `loglevel`

* add abbreviations for discovery params
2022-11-15 13:16:02 -06:00
Dmitriy Ryajov 5abf80cc69
Block download (#304)
* track inflight requests

* preperly handle precense updates

* trace number of of scheduled blocks

* invoke `payForBlocks` at the correct time

* reduntant block info on want list updates

* don't update prices in task handler

* PeerID -> PeerId

* cleanup

* proper log topic

* better chronicles topic filtering

* more trace logging

* sort want blocks

* wip - fix tests

* wip - fix tests, presence changes

* fix small test issue

* return price

* payment related changes

* misc

* re-enable payment tests

* fix warn wording

* fix `u256` conversion

* minor misc changes

* don't idle for so long on `encode`

* logging

* move buff

* disable cache by default

* disable cache by default

* fix streamOneBlock

* log node stopping/exiting

* trace logging

* don't stringify cid

* use `self`

* quick cleanup

* rename enums

* rename enums

* turns out we don't needs this test

* fix wording
2022-11-15 09:46:21 -06:00
Dmitriy Ryajov 5f9507cfcd
Blocks leak and inflight for pending blocks (#315)
* attach `wait` to handle fut

* fix crash when can't find a CID and timeout hits

the exception was not raised, thus a Nil was returned that was
than wrapped in an option, leading to crash.


Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>

* add inFlight flag

* adding pending blocks tests

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
Co-authored-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2022-11-15 00:12:05 -06:00
Dmitriy Ryajov f235cc6621
don't spawn queue requests (#313)
* don't spawn queue requests

* adjust list blocks idle timer

* increase timeout on failing test...
2022-11-14 18:01:05 -06:00
Dmitriy Ryajov f36dc54eda
Log topic filtering (#312)
* comment

* split out coders and add log level endpoin

* adjust info url

* add log filtering test

* don't stringify cid

* add log level test
2022-11-14 17:42:57 -06:00
markspanbroek 4175689745
Load purchase state from chain (#283)
* [purchasing] Simplify test

* [utils] Move StorageRequest.example up one level

* [purchasing] Load purchases from market

* [purchasing] load purchase states

* Implement myRequest() and getState() methods for OnChainMarket

* [proofs] Fix intermittently failing tests

Ensures that examples of proofs in tests are never of length 0;
these are considered invalid proofs by the smart contract logic.

* [contracts] Fix failing test

With the new solidity contracts update, a contract can only
be paid out after it started.

* [market] Add method to get request end time

* [purchasing] wait until purchase is finished

Purchase.wait() would previously wait until purchase
was started, now we wait until it is finished.

* [purchasing] Handle 'finished' and 'failed' states

* [marketplace] move to failed state once request fails

- Add support for subscribing to request failure events.
- Add supporting contract tests for subscribing to request failure events.
- Allow the PurchaseStarted state to move to PurchaseFailure once a request failure event is emitted
- Add supporting tests for moving from PurchaseStarted to PurchaseFailure
- Add state transition tests for PurchaseUnknown.

* [marketplace] Fix test with longer sleepAsync

* [integration] Add function to restart a codex node

* [purchasing] Set client address before requesting storage

To prevent the purchase id (which equals the request id)
from changing once it's been submitted.

* [contracts] Fix: OnChainMarket.getState()

Had the wrong method signature before

* [purchasing] Load purchases on node start

* [purchasing] Rename state 'PurchaseError' to 'PurchaseErrored'

Allows for an exception type called 'PurchaseError'

* [purchasing] Load purchases in background

No longer calls market.getRequest() for every purchase
on node start.

* [contracts] Add `$` for RequestId, SlotId and Nonce

To aid with debugging

* [purchasing] Add Purchasing.stop()

To ensure that all contract interactions have both a
start() and a stop() for

* [tests] Remove sleepAsync where possible

Use `eventually` loop instead, to make sure that we're
not waiting unnecessarily.

* [integration] Fix: handle non-json response in test

* [purchasing] Add purchase state to json

* [integration] Ensure that purchase is submitted before restart

Fixes test failure on slower CI

* [purchasing] re-implement `description` as method

Allows description to be set in the same module where the
state type is defined.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [contracts] fix typo

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* [market] Use more generic error type

Should we decide to change the provider type later

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-11-08 08:10:17 +01:00
markspanbroek be32b9619b
Fixes loading of private key on Windows (#299)
Unix permissions don't work on Windows; 
adds check for correct ACL settings in Windows.
2022-11-07 15:54:24 +01:00
Michael Bradley c893b1f0cb
track latest nim-libp2p's master branch (#248)
* [build] track nim-libp2p's unstable branch

* cid formatIt change

* track nim-libp2p-unstable

* don't stringify cid

* Fixed testblockexc.nim

1. Assign price to blocks
2. Delete on node1 cached blocks from node2 before buying them again

* add trace logging

* init pricing info

* remove duplicate price setting

* cid serialization in trace logs

* bumping dht to latest main

* bump

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Co-authored-by: Bulat-Ziganshin <bulat.ziganshin@gmail.com>
2022-11-02 11:40:28 -06:00
Dmitriy Ryajov 0ecbfcec9f
setup and persist private key (#292)
* setup and persist private key

* return dht record spr

* helper to remap multiaddr ip and port

* set/update discovery and announce addrs

* add nat and discovery IPs

* allow for announce and DHT addresses separatelly

* update tests

* check for nat or discoveryIp

* fix integration tests

* misc align

* don't share data dirs and and set bootstrap node

* add log scope

* remap announceAddrs after node start

* simplify discovery initialization

* make nat and disc-ip required

* add log scope don't init dht spr in constructor

* bump dht

* dissallow `0.0.0.0` for `--nat`
2022-11-01 18:58:41 -06:00
Dmitriy Ryajov ae46f4dc2f
add support for logging topics (#293)
* add support for logging topics

* add top level codex topic
2022-11-01 14:05:40 -06:00
Dmitriy Ryajov 6e4a8b86ab
Rework discovery (#288)
* use multiaddrs instead of ip/port

* rework to support updating spr

* fix tests

* fix compilation

* use base for base methods
2022-10-27 07:44:56 -06:00
Dmitriy Ryajov e50ea88411
Remove protobuf serialization (#289)
* add format for cid

* cid formatIt change

* track nim-libp2p-unstable

* rework probuf serialization for por

* add missing include

* removing nim protobuf serialization

* rollback to dht to main

* remove protobuf serialization import
2022-10-27 07:41:34 -06:00
Mark Spanbroek 92eecb0702 [purchasing] Rework purchasing as a state machine 2022-10-25 15:10:35 +11:00
Mark Spanbroek 7841a9692c [utils] Add AsyncState to state machine 2022-10-25 15:10:35 +11:00
Mark Spanbroek b52d291785 [utils] Add state machine implementation 2022-10-25 15:10:35 +11:00
Mark Spanbroek e648c26340 [utils] Add operator `as` for optional casting 2022-10-25 15:10:35 +11:00
Mark Spanbroek bc3734b1a8 [purchasing] move Purchase and PurchaseId into separate modules 2022-10-25 15:10:35 +11:00
Eric Mastro 25df599c08 Update to new reverts API, add tests
1. use `ProviderError` to catch contract errors instaed of `JsonRpcProviderError`.
2. Add tests for `isProofRequired`, `willProofBeRequired`, and `getProofEnd` when slot is empty (and is call is reverted)
3. bump nim-ethers to branch that has new `reverts` API and `ProviderError` support.
4. Update to new `reverts` API.
2022-10-25 15:10:35 +11:00
Eric Mastro 5eb65336fd catch and check revert reasons
Ensure specific contract reverts are handled and returned as booleans or default values

# Conflicts:
#	codex/contracts/proofs.nim
2022-10-25 15:10:35 +11:00
Eric Mastro 10d4dcd70d bump nim-ethers, fix tests
# Conflicts:
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro dab65eb6bd catch contract errors to return bool
# Conflicts:
#	codex/contracts/proofs.nim
#	vendor/nim-ethers
2022-10-25 15:10:35 +11:00
Eric Mastro ae9467b780 simplify getRequest 2022-10-25 15:10:35 +11:00
Eric Mastro b9e53e182f [marketplace] support freeing of slots and failed state
Support storage contract parameters to free slots after too many proofs missed, as well as fail a contract after too many slots have been freed.

# Conflicts:
#	vendor/dagger-contracts
2022-10-25 15:10:35 +11:00
Eric Mastro 8dd110765f [marketplace] Catch slot empty, return default values 2022-10-25 15:10:35 +11:00
Eric Mastro 9d218c88a5 [marketplace] Remove `isSlotCancelled` and `isCancelled`
Remove `isSlotCancelled` and `isCancelled` from proving, as it did not fit in the module.

Update the proving module to not rely on checking the contract to understand if a request was cancelled. Instead, `proofEnd` was modified in `dagger-contracts` such that it returns a past timestamp when the contract is cancelled. This successfully removes
2022-10-25 15:10:35 +11:00
Eric Mastro b4a14e00f7 [marketplace] address more PR comments
- remove RequestState as no longer being used
- remove RequestState check on purchase timeout
- add tests for `withdrawFunds` and `subscribeRequestCancelled`
- update `dagger-contracts` so that `RequestCancelled.requestId` property is indexed in the event
2022-10-25 15:10:35 +11:00
Eric Mastro 5cb74b6638 [chore] additional clean up of `array[32, byte]` types
- rename `ContractId` to `SlotId`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`
2022-10-25 15:10:35 +11:00
Eric Mastro 372f827982 [marketplace] add/remove proofs for contract state
Add or remove proof requirements when a request contract’s state changes. When a request sale has completed (for a slot), the host who purchased that slot now must provide regular proofs for the data they are contracted to hold. This is now enforced by adding the slotId to the HashSet of Ids for which to require proofs. When a request has been cancelled (not all slots were filled before the request expired), proofs no longer need to be provided and the slotId is removed from teh HashSet.

Add `isCancelled` and `isSlotCancelled` checks to query the contract state without relying the on the state context variable in the contract. Because contract state can only be updated in a transaction, and the client withdrawing funds is responsible for changing the contract state to “Cancelled”, the `isCancelled` and `isSlotCancelled` functions were introduced to check the state regardless of whether or not the client had already withdrawn their funds.
2022-10-25 15:10:35 +11:00
Eric Mastro 0c3fbad470 [purchasing] Withdraw funds when request times out
When a request for storage times out (not enough slots filled), the client will initiate a withdraw request to retrieve its funds out of the contract, setting the state of the request to RequestState.Cancelled. The client will also emit a RequestCancelled event for others to listen to (ie hosts will need to listen for this event to withdraw its collateral).

Add unit test that checks for emission of RequestCancelled after request is purchased request expires.

Update dagger-contracts dependency to commit that holds the changes supporting withdrawing of funds.
2022-10-25 15:10:35 +11:00
Bulat-Ziganshin 9939d85b74
Replace protobuf serialization for Block exchange with minprotobuf (#271)
* minprotobuf serialization for Block exchange

* Handle decoding errors by stopping peer connection
2022-10-13 18:58:57 -05:00
Bulat-Ziganshin 6e6f40016c
Changes required for reasonable upload/download speed (#265)
Increase blocksize from ~8KiB to ~64KiB and remove useless unbounded prefetching on download
* increased upload/download speed on my Win10 Zen3 box to 50/150 MB/s (with FSStore)
* made manifest files 8x smaller
* two more changes in the tests to make them pass
Closes #263
2022-10-06 23:01:25 +03:00
Dmitriy Ryajov 106b04340a
Remove local providers (#268)
* add remove provider method

* remove failed providers

* bumping dht to latest master
2022-10-05 10:01:21 -06:00
Mark Spanbroek 58f0439f37 [contracts] Use Duration instead of float for clock offset
Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-10-03 15:17:03 +02:00
Mark Spanbroek 4e6d84850a [contracts] Fix intermittently failing clock test
Use a sub-second offset to avoid off-by-one errors in the
test at the second boundary.

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
2022-10-03 15:17:03 +02:00
Dmitriy Ryajov 8d315b38b8
Dht maintenance (#267)
* protocol start is now async

* bumping to maintenance branch
2022-10-01 11:08:44 -06:00
Dmitriy Ryajov 69bd359287
Persist Dht providers (#257)
* providers store integration
2022-09-29 20:16:59 -06:00
Dmitriy Ryajov 7e2f917c84
Dht providers lru (#247)
* upgrading nimbus build system to latest

* use new `updateRecord`

* bump nim-libp2p-dht

* bump nimbus build system

* bumping deps

* update to latest dht
2022-09-13 13:37:49 -06:00
Bulat-Ziganshin df1522c9f8
Fixed message.proto to match IPFS specs (#245)
We incorrectly assigned code 2 to the payload field instead of code 3:

Our code: 8c59392526/codex/blockexchange/protobuf/message.proto (L50-L52)

While original code: 0fa397581c/message/pb/message.proto (L43-L45)

And [documentation](https://github.com/ipfs/specs/blob/main/BITSWAP.md#bitswap-120-wire-format).
2022-09-12 21:32:12 -06:00
Eric Mastro 58f3335cc1 [fix] Fix C-compilation error failing tests 2022-08-26 13:29:09 +10:00
Eric Mastro 8f116cd01e [chore] make RequestId, SlotId, Nonce, PurcahseId distinct
Make RequestId, SlotId, Nonce, PurcahseId distinct types.

Add/modify conversions to support the distinct type (ABI encoding/decoding, JSON encoding, REST decoding).

Update tests
2022-08-26 13:29:09 +10:00
Eric Mastro 6df5a7cf54 [chore] clean up `array[32, byte]` types
- rename `ContractId` to `SlotId`
- add `RequestId`, `PurchaseId`, `Nonce` types as aliases of `array[32, byte]`
- rename `Proving.contracts` to `Proving.slots`
- change signatures of `isSlotCancelled` and `isCancelled` to use `SlotId` and `RequestId` types, respectively.
- change all references to `RequestId`, `SlotId`, and `PurchaseId`
2022-08-26 13:29:09 +10:00
Bulat-Ziganshin f24ded0f76
Download files without padding (#218)
The initial goal of this patch was to allow to download of a file via REST API in exactly the same size as it was uploaded, which required adding fields Chunker.offset and Manifest.originalBytes to keep that size. On top of that, we added more integrity checks to operations on Manifest, and reorganized TestNode.nim to test the actual interaction between node.store and node.retrieve operations.

Note that the wire format of Manifest was changed, so we need to recreate all BlockStores.

* Download without padding
* Fixed chunker tests
* Chunker: get rid of RabinChunker
* Verify offset in the chunker tests
* Use manifest.originalBytesPadded in StoreStream.size
* StoreStream: replace emptyBlock with zeroMem
* Manifest.bytes: compute how many bytes corresponding StoreStream(Manifest, pad) will return
* Manifest: verify originalBytes and originalLen on new/encode/decode
Also set originalBytes in each Manifest creation/update scenario
* Manifest: comments, split code into sections
* Reordered parameters to deal with int64 size in 32-bit builds
* TestNode.nim: combine Store and Retrieve tests
1. Instead of copy-pasting code from node.nim, new test calls node.store() and node.retrieve() in order to check that they can correctly store and then retrieve data
2. New test compares only file contents, manifest contents considered an implementation detail
3. New test chunks at odd chunkSize=BlockSize/1.618 in order to ensure that data retrieved correctly even when buffer sizes mismatch
* TestNode.nim: code refactoring
* Manifest.add: one more test
* Manifest.verify: return Result instead of raising Defect
* Node.store: added blockSize parameter
2022-08-24 15:15:59 +03:00
Dmitriy Ryajov 4bc701652f
Adding metrics (#203)
* adding inflight discovery metric

* adding metrics instructions and default dashboard

* spelling fixes
2022-08-23 10:11:21 -06:00
Michael Bradley, Jr 3d823dcbc6 [stores] update getBlock return type
change return type for `method getBlock` from `Future[?!(?Block)]` to
`Future[?!Block]`

use `type BlockNotFoundError = object of CodexError` to differentiate between
"block not found in the store" and other errors

also make some logic and error handling/messages more consistent across
BlockStore implementations

closes #177
closes #182
closes #210

alternative to #205, #209
2022-08-22 20:52:51 -05:00
Bulat-Ziganshin 7ec039f822
Quickfix: start erasure decoding once any K blocks arrived (#199)
ECC decoding can be done with any encoded.K blocks, and now we start it once any K blocks arrived.

Previous code essentially waited for all blocks to arrive (or fail to arrive), that may mean unnecessary delays.

New code will sometimes ECC-decode when parity blocks were retrieved faster that data blocks (instead of waiting for data blocks from network), but we expect that it's OK due to extremely fast Leopard decoding (~~500 MB/s)
2022-08-19 21:21:55 -06:00
Michael Bradley, Jr 5aa42541b3 [build] update nim-datastore
also make changes to `codex/codex.nim` and tests for SQLiteStore re: changes in
nim-datastore

closes #178
2022-08-10 13:13:07 -05:00
markspanbroek 097a4cfd67
Better command line options for Ethereum (#181)
* [contracts] ContractInteractions.new() now requires account parameter

* [cli] Only perform Ethereum interactions when --eth-account is specified

* [cli] Add --persistence option that is disabled by default

* [cli] Use Option for ethDeployment parameter

* [node] Better error reporting when Ethereum node cannot be reached
2022-08-09 06:29:06 +02:00
Michael Bradley, Jr eebcf7e7b2 [node] custom iterator/query for SQLiteStore's `method listBlocks` 2022-08-08 17:48:37 -05:00
Michael Bradley, Jr 2ecf750959 [node] support self.cache=nil in SQLiteStore
also fix a discrepancy where cli option `--cache-size` is documented as
`0 disables the cache` in `codex/conf.nim`, but previously the value `0` would
result in a cache being constructed with default parameter values for
`CacheStore.new()`

Closes #180
2022-08-08 16:24:45 -05:00