Commit Graph

426 Commits

Author SHA1 Message Date
Mark Spanbroek afdb5be2d4 Add TODOs to indicate wanted architecture
As discussed in team:
- Retrieving of blocks that are mentioned in the
  manifest should be moved into a BlockStore.
- Sales should be given an instance of BlockStore,
  and an instance of StorageProofs, instead of the
  onStore, onClear, and onProve callbacks
2022-07-20 13:43:20 +02:00
Mark Spanbroek 13bbf2b052 Fix test that was failing intermittently
Give test more time to complete but do not increase
sleep time more than necessary, by introducing
waiting loop for test purposes.
2022-07-20 13:43:20 +02:00
Mark Spanbroek 6df0c9ab40 [integration] set expiry always to high value
Reasoning: a purchase is now completed when
the first proof comes in, so the test will no
longer wait until the request has expired.
2022-07-20 13:43:20 +02:00
Mark Spanbroek 97a5237d44 [sales] Add documentation 2022-07-20 13:43:20 +02:00
Mark Spanbroek 571aa6f694 [sales] Consistent names for callbacks 2022-07-20 13:43:20 +02:00
Mark Spanbroek 592a3dce4d [sales] Add callback for clearing storage when no longer needed 2022-07-20 13:43:20 +02:00
Mark Spanbroek adcb91a5d5 [node] Store datasets locally when sales asks for it 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7bc5280596 [node] rename PrefetchBatch -> Batch
Reason: in a next commit we'll use the same
constant to fetch data in batches
2022-07-20 13:43:20 +02:00
Mark Spanbroek d4025f4a5f [manifest] Extract logic for decoding a manifest given a cid 2022-07-20 13:43:20 +02:00
Mark Spanbroek e06211f827 [sales] rename: negotiation -> sales agent
Reason: with the new design there are no offers and
selection, making this less like a negotation, and
more like an agent trying to fulfill a storage request
2022-07-20 13:43:20 +02:00
Mark Spanbroek 96ca1de768 Cleanup: remove StorageOffer everywhere 2022-07-20 13:43:20 +02:00
Mark Spanbroek bd2fba50c7 [purchasing] Fix integration test 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4a4588e2a0 [node] Set callbacks for Sales to retrieve and prove posession of data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 94f6838d19 [node] Fix: ensure that retrieve works for small files 2022-07-20 13:43:20 +02:00
Mark Spanbroek f14eeb688b [WIP sales] Handle request expiry 2022-07-20 13:43:20 +02:00
Mark Spanbroek 9438aba5d2 [WIP sales] Handle other host fulfilling request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7f864570bd [WIP sales] call onSale() 2022-07-20 13:43:20 +02:00
Mark Spanbroek a2570f9380 [WIP sales] fulfill request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 5abd5dd674 [WIP sales] make storage available again upon failure 2022-07-20 13:43:20 +02:00
Mark Spanbroek a7f3ead44e [WIP sales] refactor tests 2022-07-20 13:43:20 +02:00
Mark Spanbroek dcb8f8a7f7 [WIP sales] Generate proof of storage 2022-07-20 13:43:20 +02:00
Mark Spanbroek 07dbb6ae18 [WIP sales] Retrieve data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 3e6d51754d [WIP sales] Make tests pass by commenting out code 2022-07-20 13:43:20 +02:00
Mark Spanbroek 0636c94b74 [market] Add market.getSigner() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 03b92a2067 [market] Add market.getHost() 2022-07-20 13:43:20 +02:00
Mark Spanbroek b414ecd67e [market] Add market.getRequest() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7c50d9f739 Update ethers to version 0.2.0 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4b5cfe8e41 [purchasing] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek b7ab9481d9 [market] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek e352a181e4 [contracts] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Ivan Yonchovski e1998e0b61
Sync submodule dependencies and lock file (#134)
- I have performed synchronization using `add-all-vendor-projects` and `nimble
lock` in `nim-workspaces` project
2022-07-19 09:31:32 -06:00
Michael Bradley, Jr c64259b0d5 ci: more comments added to `.github/workflows/ci.yml` 2022-07-06 19:03:10 -05:00
Michael Bradley, Jr 2b3195518d ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2
Main goal is to update the nim-codex CI workflow to use the
[`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the
[nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The
`msys2/setup-msys2@v2` action has been used by other Status-org projects prior
to this commit, e.g. nim-leopard, nim-datastore, nim-status.

A fix is included for the failing macOS builds, related to
[actions/virtual-environments#5819][ave5819]. See [L151][L151].

All builds [succeed][succeed] (with build arch-targets verified as far as
possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and
`windows-i386`, though the `i386` builds are presently disabled (commented
out). The `i386` builds can be enabled simply by uncommenting:

 ```
 # - os: linux
 #   cpu: i386
 ...
 # - os: windows
 #   cpu: i386
 ```

The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows
for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques
learned from developing workflows for other Status-org repos. Some comments and
code-reorg help to clarify/explain what's done in the
`Derive environment variables` step.

`-d:limitStackUsage` has been adopted for `linux-amd64` builds from
[nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been
copied into `config.nims`

`-d:limitStackUsage` can easily be dropped if it's not desirable for Codex.

Build targets use `-latest` for `runs-on`, i.e. `macos-latest`,
`ubuntu-latest`, `windows-latest`.

Through a combination of local testing and iterative pushes to GitHub, the
workflow's embedded Bash scripts have been revised to include only the
necessary steps for all builds to succeed, including `linux-i386` and
`windows-i386`.

The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed,
while coverage data generation/upload steps have been added to
`.github/workflows/ci.yml` as the final steps conditional on
`if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`.

A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the
redundant flags do not affect the build, but can be helpful when eyeballing
GitHub Actions builds with increased compile-time verbosity.

Some variable expansions used in `github/workflows/ci.yml` could result in
compilation failures if related paths include whitespace. It's not a problem
for this commit but could be a problem for a user copy-pasting from the
workflow; solving that problem is left as an exercise for the reader.

[setmsys2]: https://github.com/msys2/setup-msys2#readme
[ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml
[ave5819]: https://github.com/actions/virtual-environments/pull/5819
[L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151
[succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455
[ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159
[ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-06 19:03:10 -05:00
Michael Bradley, Jr 8f0bc54fba test: make `proc finish` async in "nodes negotiate ..." integration test
`waitFor` has been moved into the main body of integration test
`nodes negotiate ...` to reduce blockage on the main thread. That change, along
with widening the expiry in `when defined(windows)` (and bumping up
`timeout-minutes` in our CI workflow) allows Windows CI builds to succeed more
consistently, i.e. without timing out.
2022-07-06 19:03:10 -05:00
Ivan Yonchovski 19407622e7
Merge pull request #126 from yyoncho/bump-libs
Bump libbacktrace and secp256k1
2022-07-04 16:23:27 +03:00
Michael Bradley, Jr 5b34d578d8 windows: ignore *.exe 2022-06-30 17:26:24 -05:00
Michael Bradley, Jr 41bd968ac0 windows: workaround to prevent random segfaults 2022-06-30 17:26:24 -05:00
Bulat-Ziganshin 15ae157e61
BlockStore: delBlock got new type (#123)
Now every store traces only its own operations and failures (currently implemented only for delBlock)
2022-06-28 19:10:05 +03:00
Ivan Yonchovski 9c2224e306 Bump libbacktrace and secp256k1 2022-06-28 11:04:18 +03:00
Dmitriy Ryajov c0b385176d
first pass over the readme (#116)
* first pass over the readme

* wording&typos

* more wording

* add multiaddr link and wording fixes

* add `make update` to instructions

* fix upload command

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* fix spacing

* adding review suggestion

* more review suggestions

* adding discord badge

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* fix levels

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-06-23 15:45:42 -06:00
Michael Bradley 2f475115d6
Merge pull request #124 from status-im/build/msys2_fix
build: override CMAKE_ARGS for Windows+MSYS2 environments re: libbacktrace
2022-06-23 16:27:10 -05:00
Michael Bradley, Jr 54616ab727
build: override CMAKE_ARGS for Windows+MSYS2 environments re: libbacktrace
Otherwise `-G"MinGW Makefiles"` is included in `CMAKE_ARGS`, which does not
function correctly in MSYS2 Bash.
2022-06-23 14:11:13 -05:00
Ivan Yonchovski 1088e43c2d
Fix command line args setup (#120)
- commandLineParams is the correct way to access args
2022-06-22 15:37:05 -06:00
Dmitriy Ryajov cf4a0a393e
Json info (#115)
* make info return a json with an spr

* add an `exec` target to the makefile
2022-06-14 17:34:56 -06:00
Dmitriy Ryajov 28ebb16767
adding AsyncStreamWrapper (#110)
* Adding AsyncStreamWrapper type

* use asyncstreamwrapper to avoid copies

* make size an operator

* export asyncstreamwrapper
2022-06-14 09:19:35 -06:00
Dmitriy Ryajov 77733725bf
fix failing test after nim-ethers bump (#112) 2022-06-09 09:36:59 -06:00
Eric Mastro 3b0e93eab4
[chore] bump nim-ethers to 0.1.7 (#107)
* [chore] bump nim-ethers to 0.1.7

* [chore] make block handler callback async

nim-ethers 0.1.7 makes the subscription callbacks async.
2022-06-02 16:09:53 -06:00
Dmitriy Ryajov 6ce7e23767
Upload authenticators (#108)
* initial implementation of storage proofs upload

* make sure proof verifies with after deserializing

* add por store

* rename por store to stp store

* rename porstore to stpstore

* add support for host discovery to discovery mock

* add tags upload network tests
2022-05-25 20:29:31 -06:00
Dmitriy Ryajov 738738c3c6
Adding storage proofs store (#109)
* add por store

* rename por store to stp store

* add stp tests
2022-05-24 18:42:49 -06:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00