Commit Graph

466 Commits

Author SHA1 Message Date
Eric 37b3d99c3d
Improve integration testing client (CodexClient) and json serialization (#514)
* Improve integration testing client (CodexClient) and json serialization

The current client used for integration testing against the REST endpoints for Codex accepts and passes primitive types. This caused a hard to diagnose bug where a `uint` was not being deserialized correctly.

In addition, the json de/serializing done between the CodexClient and REST client was not easy to read and was not tested.

These changes bring non-primitive types to most of the CodexClient functions, allowing us to lean on the compiler to ensure we're providing correct typings. More importantly, a json de/serialization util was created as a drop-in replacement for the std/json lib, with the main two differences being that field serialization is opt-in (instead of opt-out as in the case of json_serialization) and serialization errors are captured and logged, making debugging serialization issues much easier.

* Update integration test to use nodes=2 and tolerance=1

* clean up
2023-09-01 15:44:41 +10:00
Slava fc3eac9dbc
Use latest tag for Tags and Default branch only (#540) 2023-08-30 13:09:52 +03:00
Adam Uhlíř 098baf47d1
chore: improve logging (#526) 2023-08-28 11:33:44 +02:00
Slava 41cb658c1a
Update badges (#533)
* Update badges

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Add Docker build status badge

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

* Update CI and Docker badges links

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>

---------

Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-08-27 07:22:09 +03:00
Ben Bierens f10c3f3934
Debug/extra logging (#509)
* logs

* poll log

* more logs

* time log for upload steps

* Adds metric and log for block retrieval time

* adds some logging

* move logging for storestream block indices

* Log at start and end of block iteration

* applies branch blockstore-bugfix

* Cleanup

* Cleanup
2023-08-22 08:35:16 +02:00
Ben Bierens 238771eb73
Repostore crash debug (#521)
* repostore listBlocks reads too much data

* Add assert to make sure data is not queried
2023-08-22 08:26:58 +02:00
Adam Uhlíř f459a2c6f6
refactor: merging proving module into sales (#469)
Co-authored-by: Eric <5089238+emizzle@users.noreply.github.com>
2023-08-21 12:26:43 +02:00
Eric 9cecb68520
[repostore] Retrieve empty blocks (#513)
Add handling of empty blocks in the RepoStore.

* Add empty block handling to repostore for put, del, has
Also added tests for all empty block handling blockstore operations. This showed there was an ambiguous identifier present for `hasBlock`, so one of the two `hasBlock` definitions was removed in `repostore`.

* Change CacheStore to RepoStore in testerasure
As CacheStore is not used in the node, update the Datastore used in the erasure coding tests to be a RepoStore. This ensures that the K > 1 cases are being tested, where they will produce empty padding blocks in the erasure-coded manifests.
2023-08-21 12:51:04 +10:00
Slava ba41b5a232
Add reusable workflow for Docker (codex-storage/cs-codex-dist-tests/issues/42) (#518) 2023-08-17 06:04:50 +03:00
Tomasz Bekas e8601274b9
Merkle tree construction (#504)
* Building a merkle tree

* Obtaining merkle proof from a tree

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-08-15 13:23:35 +02:00
Adam Uhlíř 39efac1a97
fix: load slots on sales module start (#510) 2023-08-15 11:39:49 +02:00
Jaremy Creechley 7efa9177df
Bump deps take2 (#492)
* extra utilities and tweaks
* add atlas lock
* update ignores
* break build into it's own script
* update url rules
* base off codexdht's
* compile fixes for Nim 1.6.14
* update submodules
* convert mapFailure to procs to work around type resolution issues
* add toml parser for multiaddress
* change error type on keyutils
* bump nimbus build to use 1.6.14
* update gitignore
* adding new deps submodules
* bump nim ci version
* even more fixes
* more libp2p changes
* update keys
* fix eventually function
* adding coverage test file
* move coverage to build.nims
* use nimcache/coverage
* move libp2p import for tests into helper.nim
* remove named bin
* bug fixes for networkpeers (from Dmitriy)

---------

Co-authored-by: Dmitriy Ryajov <dryajov@gmail.com>
2023-08-01 16:47:57 -07:00
Eric 3e80de3454
Cancel and wait for asyncstatemachine futures when stopping (#493)
* Simplify `.then` (promise api) and tests

* Remove tracked future when cancelled. Add tracked future tests

* Track and cancel statemachine futures

The futures created in each asyncstatemachine instance are tracked, and each future is cancelled and waited in `stop`.

Change `asyncstatemachine.stop` to be async so `machine.trackedFutures.cancelAndWait` could be called.
Add a constructor for `asyncstatemachine` that initialises the `trackedFutures` instance, and call the constructor from derived class constructors.
2023-07-31 15:09:34 +10:00
Eric 1d161d383e
Slot queue (#455)
## Slot queue
Adds a slot queue, as per the [slot queue design](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#slot-queue).

Any time storage is requested, all slots from that request are immediately added to the queue. Finished, Canclled, Failed requests remove all slots with that request id from the queue. SlotFreed events add a new slot to the queue and SlotFilled events remove the slot from the queue. This allows popping of a slot each time one is processed, making things much simpler.

When an entire request of slots is added to the queue, the slot indices are shuffled randomly to hopefully prevent nodes that pick up the same storage requested event from clashing on the first processed slot index. This allowed removal of assigning a random slot index in the SalePreparing state and it also ensured that all SalesAgents will have a slot index assigned to them at the start thus the removal of the optional slotIndex.

Remove slotId from SlotFreed event as it was not being used. RequestId and slotIndex were added to the SlotFreed event earlier and those are now being used

The slot queue invariant that prioritises queue items added to the queue relies on a scoring mechanism to sort them based on the [sort order in the design document](https://github.com/codex-storage/codex-research/blob/master/design/sales.md#sort-order).

When a storage request is handled by the sales module, a slot index was randomly assigned and then the slot was filled. Now, a random slot index is only assigned when adding an entire request to the slot queue. Additionally, the slot is checked that its state is `SlotState.Free` before continuing with the download process.

SlotQueue should always ensure the underlying AsyncHeapQueue has one less than the maximum items, ensuring the SlotQueue can always have space to add an additional item regardless if it’s full or not.

Constructing `SlotQueue.workers` in `SlotQueue.new` calls `newAsyncQueue` which causes side effects, so the construction call had to be moved to `SlotQueue.start`.

Prevent loading request from contract (network request) if there is an existing item in queue for that request.

Check availability before adding request to queue.

Add ability to query market contract for past events. When new availabilities are added, the `onReservationAdded` callback is triggered in which past `StorageRequested` events are queried, and those slots are added to the queue (filtered by availability on `push` and filtered by state in `SalePreparing`).

#### Request Workers
Limit the concurrent requests being processed in the queue by using a limited pool of workers (default = 3). Workers are in a data structure of type `AsyncQueue[SlotQueueWorker]`. This allows us to await a `popFirst` for available workers inside of the main SlotQueue event loop

Add an `onCleanUp` that stops the agents and removes them from the sales module agent list. `onCleanUp` is called from sales end states (eg ignored, cancelled, finished, failed, errored).

Add a `doneProcessing` future to `SlotQueueWorker` to be completed in the `OnProcessSlot` callback. Each `doneProcessing` future created is cancelled and awaited in `SlotQueue.stop` (thanks to `TrackableFuturees`), which forced `stop` to become async.
  - Cancel dispatched workers and the `onProcessSlot` callbacks, prevents zombie callbacks

#### Add TrackableFutures
Allow tracking of futures in a module so they can be cancelled at a later time. Useful for asyncSpawned futures, but works for any future.

### Sales module
The sales module needed to subscribe to request events to ensure that the request queue was managed correctly on each event. In the process of doing this, the sales agents were updated to avoid subscribing to events in each agent, and instead dispatch received events from the sales module to all created sales agents. This would prevent memory leaks on having too many eventemitters subscribed to.
  - prevent removal of agents from sales module while stopping, otherwise the agents seq len is modified while iterating

An additional sales agent state was added, `SalePreparing`, that handles all state machine setup, such as retrieving the request and subscribing to events that were previously in the `SaleDownloading` state.

Once agents have parked in an end state (eg ignored, cancelled, finished, failed, errored), they were not getting cleaned up and the sales module was keeping a handle on their reference. An `onCleanUp` callback was created to be called after the state machine enters an end state, which could prevent a memory leak if the number of requests coming in is high.

Move the SalesAgent callback raises pragmas from the Sales module to the proc definition in SalesAgent. This avoids having to catch `Exception`.
  - remove unneeded error handling as pragmas were moved

Move sales.subscriptions from an object containing named subscriptions to a `seq[Subscription]` directly on the sales object.

Sales tests: shut down repo after sales stop, to fix SIGABRT in CI

### Add async Promise API
  - modelled after JavaScript Promise API
  - alternative to `asyncSpawn` that allows handling of async calls in a synchronous context (including access to the synchronous closure) with less additional procs to be declared
  - Write less code, catch errors that would otherwise defect in asyncspawn, and execute a callback after completion
  - Add cancellation callbacks to utils/then, ensuring cancellations are handled properly

## Dependencies
- bump codex-contracts-eth to support slot queue (https://github.com/codex-storage/codex-contracts-eth/pull/61)
- bump nim-ethers to 0.5.0
- Bump nim-json-rpc submodule to 0bf2bcb

---------

Co-authored-by: Jaremy Creechley <creechley@gmail.com>
2023-07-25 12:50:30 +10:00
Ben Bierens 14c5270e83
Add metrics (#478)
* Adds metrics to block exchange.

* Adds metrics to purchasing

* Adds metrics to upload and download API

* Adds metrics to the repo store

* Fixes exception in repostore start.

* Merge managed to mess up indentation.
2023-07-20 09:56:28 +02:00
Jaremy Creechley 8bebc90428
split windows CI tests into parts (#485)
* setup to use env variable for parallel tests
* use explicit targets
* use target in includes
* test windows split
* try simpler logic
* only use include in ci.yml
* add cpu
* fix name
* re-add unit tests!
* restore coverage
* not sure why env broke there
* startup node for part 1 & 2 tests
* fixup part 1 & 2 tests
* split windows into 3!
2023-07-19 16:00:34 -07:00
Tomasz Bekas 113f7cccc1
Fix a typo in docs/TWOCLIENTTEST.md (#470) 2023-07-19 17:35:16 +02:00
Ben Bierens 6708202a5f
Makes rootHash private to Manifest object. (#488)
* Makes rootHash private to Manifest object.

* Locks down all fields of Manifest.

* Review comments by Mark
2023-07-19 16:06:59 +02:00
Tomasz Bekas 3bb5960739
Fix for decoding large manifests (#479) 2023-07-19 15:10:14 +02:00
Ben Bierens 7227a4a38d
Restores the three key lines that make peer discovery work. (#484) 2023-07-18 07:51:20 +02:00
jessiebroke ac2aca42b1
Block exchange debug (#426)
* Adds sequence diagram for local download

* Adds network download flow

* makes node properly request block when presence-list is received

* starting to work :O

* Fixes fully connected download test up to 20 nodes.

* Better sequence diagram of network block resolution

* Cleans up comments and traces.

* Accidentally deleted important else

* Review comments by Dmitriy

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-07-18 07:50:47 +02:00
markspanbroek a89938478a
[integration] fix ci: longer sleeps in test (#482) 2023-07-13 15:58:35 +02:00
markspanbroek 3879ec8e3a
[clock] waitUntil() completes immediately when block arrives (#475)
Previously it could take up to one second to complete
the future. This messed with the timings in the
integration tests and made them less predictable.
2023-07-13 11:19:45 +02:00
Slava 6dd7e55719
Enable codex_enable_api_debug_peers (#471) (#474)
- Enable codex_enable_api_debug_peers

 - Add NAT_IP_AUTO for Dist-Tests
2023-07-10 16:15:06 +03:00
Jaremy Creechley 711e5e09d1
Plumbing in conf types (#472)
Goal is to provide documentation and to enable conf utils byte params.

* Create byte units NByte and plumb through in size locations.
* add json serde
* plumb parseDuration to conf
2023-07-06 16:23:27 -07:00
Slava f053135f68
feat: add docker auto-builds (#461)
https://github.com/codex-storage/nim-codex/issues/412
2023-07-04 14:46:49 +03:00
Slava 04ea8c36bf
Update CI workflow trigger branch (#464)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-07-03 17:12:21 +03:00
Tomasz Bekas 22f7bd0ea9
Tests for AsyncStreamWrapper (#463) 2023-06-30 14:18:58 +02:00
Slava e0d127f1d5
Add Feature request template (#465) 2023-06-29 08:44:04 +03:00
Dmitriy Ryajov bd594c9aaf
Create memory-leak detecting test suite (#226)
* adding tracker for streamstore

* adding tracker tests

* Sets up tracker helper functions and closes streams in testnode.nim

* Deploying checksuite for memory leak tracker checking.

* Successfully deploys checksuite and asyncchecksuite.

* Fix leak in testpor.nim

* Fixes leaked storestream in testnetwork.nim

* Fixes integration tests

* Cleanup

* cleanup comment by Mark

---------

Co-authored-by: benbierens <thatbenbierens@gmail.com>
2023-06-22 12:01:21 -06:00
Jaremy Creechley e47b38af11
Improving proc/func formatting consistency (#454)
* Fixes/workarounds for nimsuggest failures in codex.nim.

* remove rng prefix - it appears to work now

* format new's to be more consistent

* making proc formatting a bit more consistent
2023-06-22 08:11:18 -07:00
Eric Mastro 2a92dc9702
[marketplace] Simulate invalid proof submissions (#393) 2023-06-22 20:32:18 +10:00
Jaremy Creechley 6d1469b4be
Nimsuggest tweaks (#453)
* Fixes/workarounds for nimsuggest failures in codex.nim.
* remove rng prefix - it appears to work now
2023-06-21 15:02:05 -07:00
Adam Uhlíř cfd2cf9302
feat: bigint uses decimal over hex encoding (#452) 2023-06-21 07:46:18 +02:00
Adam Uhlíř 4cd8dd2e05
feat: slots rest api (#443)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-06-20 14:52:15 +02:00
Ben Bierens 219c7704b9
Dist-test failure due to argument name change (#451) 2023-06-19 15:35:13 +02:00
Adam Uhlíř 66b5f23b65
chore: remove unused imports (#445) 2023-06-19 11:53:41 +02:00
Ben Bierens db7c4b5811
Fixes docker image. (#448) 2023-06-19 11:00:21 +02:00
Ben Bierens 8c232b6759
Docker multiarch debug (#447)
* Uses correct string for marketplace address

* first steps towards support for arm64

* Applies multiarch ubuntu dockerfile as codex-dockerfile.

* Add `--simulate-proof-failures` env variable, update docker-compose to point to slimmed image

* Add image to CI, and update startCodex.sh

* Sets up separate docker build for arm

* Update arm64 arm of docker ci

* [docker] modify startCodex.sh

- include overridden node name in log output if specified in test
- quote `—log-level` value so that multiple log levels can be specified
- ensure any CLI parameter env vars are passed through to the codex binary, instead of conditionally including them
- add `—persistence`
- add `—validator`

* fixes load and push for amd docker build

---------

Co-authored-by: Eric Mastro <github@egonat.me>
2023-06-19 08:28:27 +02:00
Ben Bierens db9d90b465
Dht debug (#446)
* Support for building docker images with local modifications for the purpose of testing and debugging

* exposes peer information via debug/info

* api-names slightly kinder to json serializers

* Moves debug image-building

* fixes misalignment of debug peer info array

* Changes switchPeers source from KeyBook to AddressBook (filed ticket in libp2p, discussed with Tanguy)

* Limited success with dist-test peer discovery tests

* Removes unnecessary random-timer

* bumps dht repo. Adds peerId to formatNode

* Removes unused prints

* bumps libp2p-dht

* Exposes node address on debug api

* Adds traces

* review comments by me

* Hides debug/peers api behind compile flag

* Waiting for nim-libp2p-dht PR merge

* bumps libp2p-dht back to main after PRs were merged there.

* Cleanup
2023-06-19 08:21:03 +02:00
Adam Uhlíř 4f99d88c0a
feat: allow any case for logLevel (#444) 2023-06-16 18:20:49 +02:00
Adam Uhlíř 720c372be0
fix(api): set content type for json (#441) 2023-06-13 15:33:01 +02:00
Slava cfe9225be1
Fix Open API workflow branch (codex-storage/infra-codex/issues/5) (#442)
Signed-off-by: Slava <20563034+veaceslavdoina@users.noreply.github.com>
2023-06-12 12:34:44 +03:00
markspanbroek 3181361658
Cleanup purchasing state machine (#422)
* [state machine] Allow querying of state properties

* [purchasing] use new state machine

* [state machine] remove old state machine implementation

* [purchasing] remove duplication in error handling
2023-06-05 10:48:06 +02:00
Slava 3e7ce137a4
Update links to codex-storage organization (#420) 2023-05-23 23:01:13 +03:00
Adam Uhlíř c075890737
feat: storage host collects reward (#416)
Co-authored-by: markspanbroek <mark@spanbroek.net>
2023-05-15 09:02:57 +02:00
Adam Uhlíř 4d028c6cb3
feat: contract address management (#405)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-05-03 09:24:25 +02:00
Adam Uhlíř ff6cc01857
feat: adding config and env. variable support (#379)
Co-authored-by: Eric Mastro <github@egonat.me>
2023-05-02 15:06:34 +02:00
markspanbroek 8cfac5fee8
[validator] fix intermittent test failure (#402)
Test could fail when it was started right at the
boundary of a period (on the system clock)
2023-05-01 17:22:13 +02:00
markspanbroek 515304f9c5
[node] fix: stop validator (#401) 2023-05-01 16:23:26 +02:00