Commit Graph

533 Commits

Author SHA1 Message Date
Eric 34c4fb07fe
add openapi spec for /sales/slots/{slotId} 2023-12-06 15:40:47 +11:00
Eric 658302802f
Use compile flag to determine if the latest block should be used for clock.now
Due to a bug in hardhat, the latest block timestamp was out of sync from the newHeads block timestamp which was causing some timing issues in the tests. To fix this, a compile flag, `codex_testing` (which was changed from `codex_enable_proof_failures`) is used to determine the bevhaviour of clock.now. If `codex_testing` is true, clock.now uses the latest block timestamp, else it uses the cached block.timestamp that was populated in the `newHeads` event.
2023-12-06 15:40:47 +11:00
Eric c19a94ddc1
move node config from suite-level to test-level 2023-12-06 15:40:47 +11:00
Eric ec4f5c015f
clean up unused import warnings 2023-12-06 15:40:47 +11:00
Eric c9b6bfb2a7
clean up 2023-12-06 15:40:47 +11:00
Eric f95fa0a97c
fix partial payout integration test
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
2023-12-06 15:40:46 +11:00
Eric cbd357e830
clean up 2023-12-06 15:39:08 +11:00
Eric e762860e6b
speed up tests
- advance to next period and mine every 500ms
- update eventually checks in tests to check for number of periods, not time
- change log file format to include test name
2023-12-06 15:39:08 +11:00
Eric 045f770128
All tests working, with no time advancement
- works with hardhat already running
- works with ethersuite to revert snapshot (ethProvider instead of provider)
- OnChainClock.now() gets latest block timestamp
2023-12-06 15:39:08 +11:00
Eric 3d9eb19855
All integration tests working 2023-12-06 15:39:08 +11:00
Eric ca4181fe00
WIP proofs tests 2023-12-06 15:39:08 +11:00
Eric 93f9dbba08
WIP getting integration tests to work with marketplacesuite 2023-12-06 15:39:08 +11:00
Eric eda4a85c3e
DRYing, renaming 2023-12-06 15:39:08 +11:00
Eric 935929461f
fix hardhat stop
- allow multiple multinode suites in one module
- remove need to rely on npmstart script, use eval command instead
- properly override stop method to shutdown hardhat node
2023-12-06 15:39:08 +11:00
Eric 4dffb643a4
comment out other integration tests for now 2023-12-06 15:39:08 +11:00
Eric 3964f868f5
Remove cancelTransaction, add period cadence to testproofs 2023-12-06 15:39:08 +11:00
Eric bc0652aeb6
Get balance for each provider, bump ethers 2023-12-06 15:39:08 +11:00
Eric 10decd2f28
allow integration tests to start hardhat locally so logs can be captured 2023-12-06 15:39:08 +11:00
Eric 40ee931d6d
remove cancellation of txs in market, handled by ethers now 2023-12-06 15:39:08 +11:00
Adam Uhlíř 3b520fc0c6
fix: unknown state goes to payout when slot state is finished (#555)
Support logging to file

Log the entire config and fix build error

Process slot queue on reservation callback onMarkUnused

Add Reservation object, rename reserve > create

refactor Reservations api to include Reservation CRUD

All tests that use the Reservation module updated

- add requestId and slotIndex to Reservation (hopefully these will prove to be useful when we persist Reservations until request are completed, to add back bytes to Availability)
- add querying of all reservations, with accompanying tests
- change from find to findAvailabilities
- move onCleanUp from SalesContext to SalesAgent as it was getting overwritten for each slot processed
- remove sales agent AFTER deleting reservation, as this was causing some SIGSEGVs
- retrofit testsales and testslotqueue to match updated Reservations module API

Add deletion of inactive reservations on sales load

clean up

add exception message detail util

Apply to onStore errors as we are seeing undetailed errors in the dist tests logs

add missing file

change slotsize to reflect current implemenation

Fix slotSize to reduce by one block

Revert change to slotSize that reduces it by one block

Add additional test check for querying reservations/availabilities

filter past requests based on availability

Because availability filtering on push was removed, when availability is added and past storage request events are queried, those requests need to be filtered by availability before being added to the queue.

Revert "filter past requests based on availability"

This reverts commit 0c2362658b.

Add debugging for dist tests

Add cancel on error during filling state

When calling fillSlot, any transaction errors that occur (possibly during estimate gas) will cause that tx to be replaced with a cancellation transaction (a 0-valued tx to ourselves).

more debug logging

fix build

wait for fillSlot to be mined (so that a revert error can be extracted)

fix confirmation of fillSlot

add more debugging

moar debugging

debugging: change echo to trace

bump ethers to add chronicles

fix contracts tests

switch to earlier nim-ethers commit

bump json-rpc and nim-ethers

bump nim-ethers to prevent parsing newHeads log from crashing sales state machine

moar debugging

moar debugging

moar debugging

bump ethers to fix "key not found: data" error

ethers debug logging

bump ethers - better Transaction object deserialization

bump ethers to replay tx with past tx format

bump ethers to add serialization for PastTransaction

ethers bump: better logging, separate logic for revert reason string

fix build

ethers: update revert reason retreival

remove unneeded confirm

ethers: try replay without decrementing blockNumber

ethers: include type and chainId in replayed txs

ethers: add gas into and remove type from replayed txs

ensure gas is being serialized in Transaction

ethers: fix build error

bump ethers: rebased on top of cancel tx due to estimateGas error PR

Fix proving with latest ethers bump (tested on master)

Update multinode suite for better simulateFailedProofs enabling, add proofs test

Improve multinode suite for better debug options, including logging to file

There is a 503 "sales unavailable" error

improve multinode test suite, add logging to file
2023-12-06 15:39:03 +11:00
Eric 86a6ef9215
remove waitFor cancelAndWait in proving state (#643)
In proving.onCancelled, the `waitFor state.loop.cancelAndWait()` was never completing. Turns out this was not needed, because when changing states, the current state's run is cancelled, which automatically cancels the state prove loop, because it is a child of proving.run. Therefore, the logic to cancelAndWait the prove loop was removed as it was not needed.
2023-12-06 14:20:14 +11:00
Adam Uhlíř b38146d3f7
fix: check expiration is before request end (#641) 2023-12-05 14:25:28 +01:00
Adam Uhlíř 0e4387d1b3
refactor: move expiry update from fetchBatched (#634) 2023-11-28 22:04:11 +01:00
Dmitriy Ryajov 22c31046a7
Dont dial self (#633)
* don't dial self

* revert style changes

* filter out self on dial

* add helpers to `peerId` and `isSelf`

* don't fire up discovery eaguerly

* allow excluding multiple peers in sendWantHave

* revert style changes

* move self check to discovery.find

* readd eaguer dht lookup is required in some cases

* revert style changes

* misc

* drop peer first, before queueing a dht lookup

* moar style changes

* use isSelf
2023-11-27 10:25:53 -08:00
Dmitriy Ryajov e871859a96
Bump dht (#628)
* rename nim-libp2p-dht to nim-codex-dht

* bump dht

* update .gitmodules

* bump

---------

Co-authored-by: Jaremy Creechley <creechley@gmail.com>
2023-11-27 09:23:25 -08:00
markspanbroek b62ccf5a8a
Update Request: remove PoR, add merkle root (#590)
* [build] update codex-contracts-eth

* [contracts] update request parameters

- Remove PoR parameters and totalChunks
- Add merkleRoot
2023-11-27 12:25:01 +00:00
Slava a805af02c1
Run C-Tests on master commits (#636)
https://github.com/codex-storage/cs-codex-dist-tests/pull/84
2023-11-27 13:32:13 +02:00
Adam Uhlíř 79fce39dbf
fix: require expiry from api (#629)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 11:35:26 +00:00
Adam Uhlíř 8681a40ee7
feat: update expiry when data downloaded and slot filled (#619)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-11-22 10:09:12 +00:00
Tomasz Bekas 4d546f9ace
Bump questionable and test for tree init failure (#630) 2023-11-21 19:30:14 +01:00
Dmitriy Ryajov ec8d0c98b2
Fix REST endpoints semantics (#612)
* Fix REST endpoints semantics

* update endpoint description

* update, operation id

* Adding enum support

* make enum descerializer public

* add support for listing manifests

* test `/data` endpoint to list local manifests

* debug leftovers

* remove commented out line
2023-11-20 16:14:06 -08:00
Tomasz Bekas 70efd13c73
Fix block delete by treeCid and index (#623) 2023-11-20 14:48:56 +01:00
markspanbroek 374e6b645b
Add poseidon2 dependency (#625)
* [build] add nim-poseidon2

* [build] Workaround incompatibility constantine and secp256k1

Constantine sets compiler flag that is incompatible
with the assembly code in secp256k1:
5f7ba18f2e/constantine/platforms/isa/macro_assembler_x86.nim (L19)

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-20 09:51:36 +01:00
Ben Bierens 750fe2392f
Adds option to automatically announce codex by its public IP (#621)
* Adds option to automatically announce codex by its public IP

* Review comments by Slava

* Make NAT_PUBLIC_IP_AUTO more verbose and error resistant

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
Co-authored-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-11-17 13:48:25 +00:00
Ben Bierens bece1b88a1
Feat/bump questionable (#627)
* Bumps questionable version to 0.10.12

* removes unnecessary questionnable bindings.

* Fixes tests

* unnecessary whitespaces
2023-11-17 13:49:45 +01:00
Ben Bierens 40d7714a67
Fixes json formatting for NodeId (#614) 2023-11-15 13:29:17 +00:00
Dmitriy Ryajov 4d50ecf504
handle exceptions gracefuly in rest api (#617) 2023-11-14 11:05:43 -08:00
Dmitriy Ryajov 8d61391073
Light cleanup (#616)
mostly whitespaces cleanup
2023-11-14 09:52:27 -08:00
Dmitriy Ryajov 06bb21bfc7
backport changes from fix-erasure-tests (#618) 2023-11-14 08:53:06 -08:00
Dmitriy Ryajov a1725594a8
Fix json serialization for cid and enums (#615)
* use str on JString types, `$` will preserve `"`

* Adding enum support

* deserialize cid test

* make enum descerializer public

* unify fromJson for objects and refs

* add enum descerialization testing
2023-11-14 07:23:50 -08:00
Tomasz Bekas 2396c4d76d
Blockexchange uses merkle root and index to fetch blocks (#566)
* Blockexchange uses merkle root and index to fetch blocks

* Links the network store getTree to the local store.

* Update codex/stores/repostore.nim

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>

* Rework erasure.nim to include recent cleanup

* Revert accidential changes to lib versions

* Addressing review comments

* Storing proofs instead of trees

* Fix a comment

* Fix broken tests

* Fix for broken testerasure.nim

* Addressing PR comments

---------

Signed-off-by: Tomasz Bekas <tomasz.bekas@gmail.com>
Co-authored-by: benbierens <thatbenbierens@gmail.com>
Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-11-14 13:02:17 +01:00
Adam Uhlíř 778093d12e
docs(api): add note about nodes in request storage creation (#547) 2023-11-13 14:39:22 +00:00
Ben Bierens 34a9c5e88d
Storage/purchases endpoint (#609)
* adds list of purchase ids to storage API

* adds list of purchase ids to storage API

* Updates openAPI

* fixes openAPI type
2023-11-13 11:30:27 +00:00
Ben Bierens 80e2ef4eb5
Debug/info shows announce addresses (#610)
* Shows announce addresses in debug/info endpoint.

* Shows announce addresses in debug/info endpoint.
2023-11-12 09:58:40 +01:00
Ben Bierens cb02962231
Adds endpoint for listing files (manifests) in node. Useful for demo UI. (#599)
* Adds endpoint for listing files (manifests) in node. Useful for demo UI.

* Moves upload/download/files into content API calls.

* Cleans up json serialization for manifest

* Cleans up some more json serialization

* Moves block iteration and decoding to node.nim

* Moves api methods into their own init procs.

* Applies RestContent api object.

* Replaces format methods with Rest objects in json.nim

* Unused import

* Review comments by Adam

* Fixes issue where content/local endpoint clashes with content/cid.

* faulty merge resolution

* Renames content API to data.

* Fixes faulty rebase

* Adds test for data/local API

* Renames local and download api.
2023-11-09 08:47:09 +00:00
Eric 7d4ea878d2
Support logging to file (#558)
* Support logging to file

* Log the entire config and fix build error

* Downgrade log level for "starting codex node" config output

* bump ethers to prevent nonce gaps

* fix tests
2023-11-09 16:35:55 +11:00
Adam Uhlíř c0bec2f899
feat: ensure block expiry (#597)
* feat: update block expiry

* chore: feedback implementation

* chore: feedback implementation

* chore: feedback implementation
2023-11-06 08:10:30 +00:00
Slava 11cd2c46ad
Rename metrics (#478) (#603) 2023-11-03 17:21:54 +02:00
Adam Uhlíř 0014ffdef5
test: integration tests listen on localhost (#596) 2023-10-24 16:52:06 +02:00
Adam Uhlíř 2fc71cf81b
feat: partial payouts for cancelled requests (#561) 2023-10-24 10:12:54 +00:00