Commit Graph

571 Commits

Author SHA1 Message Date
Mark Spanbroek 592a3dce4d [sales] Add callback for clearing storage when no longer needed 2022-07-20 13:43:20 +02:00
Mark Spanbroek adcb91a5d5 [node] Store datasets locally when sales asks for it 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7bc5280596 [node] rename PrefetchBatch -> Batch
Reason: in a next commit we'll use the same
constant to fetch data in batches
2022-07-20 13:43:20 +02:00
Mark Spanbroek d4025f4a5f [manifest] Extract logic for decoding a manifest given a cid 2022-07-20 13:43:20 +02:00
Mark Spanbroek e06211f827 [sales] rename: negotiation -> sales agent
Reason: with the new design there are no offers and
selection, making this less like a negotation, and
more like an agent trying to fulfill a storage request
2022-07-20 13:43:20 +02:00
Mark Spanbroek 96ca1de768 Cleanup: remove StorageOffer everywhere 2022-07-20 13:43:20 +02:00
Mark Spanbroek bd2fba50c7 [purchasing] Fix integration test 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4a4588e2a0 [node] Set callbacks for Sales to retrieve and prove posession of data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 94f6838d19 [node] Fix: ensure that retrieve works for small files 2022-07-20 13:43:20 +02:00
Mark Spanbroek f14eeb688b [WIP sales] Handle request expiry 2022-07-20 13:43:20 +02:00
Mark Spanbroek 9438aba5d2 [WIP sales] Handle other host fulfilling request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7f864570bd [WIP sales] call onSale() 2022-07-20 13:43:20 +02:00
Mark Spanbroek a2570f9380 [WIP sales] fulfill request 2022-07-20 13:43:20 +02:00
Mark Spanbroek 5abd5dd674 [WIP sales] make storage available again upon failure 2022-07-20 13:43:20 +02:00
Mark Spanbroek a7f3ead44e [WIP sales] refactor tests 2022-07-20 13:43:20 +02:00
Mark Spanbroek dcb8f8a7f7 [WIP sales] Generate proof of storage 2022-07-20 13:43:20 +02:00
Mark Spanbroek 07dbb6ae18 [WIP sales] Retrieve data 2022-07-20 13:43:20 +02:00
Mark Spanbroek 3e6d51754d [WIP sales] Make tests pass by commenting out code 2022-07-20 13:43:20 +02:00
Mark Spanbroek 0636c94b74 [market] Add market.getSigner() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 03b92a2067 [market] Add market.getHost() 2022-07-20 13:43:20 +02:00
Mark Spanbroek b414ecd67e [market] Add market.getRequest() 2022-07-20 13:43:20 +02:00
Mark Spanbroek 7c50d9f739 Update ethers to version 0.2.0 2022-07-20 13:43:20 +02:00
Mark Spanbroek 4b5cfe8e41 [purchasing] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek b7ab9481d9 [market] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Mark Spanbroek e352a181e4 [contracts] Update to latest dagger-contracts 2022-07-20 13:43:20 +02:00
Ivan Yonchovski e1998e0b61
Sync submodule dependencies and lock file (#134)
- I have performed synchronization using `add-all-vendor-projects` and `nimble
lock` in `nim-workspaces` project
2022-07-19 09:31:32 -06:00
Michael Bradley, Jr c64259b0d5 ci: more comments added to `.github/workflows/ci.yml` 2022-07-06 19:03:10 -05:00
Michael Bradley, Jr 2b3195518d ci: update GitHub Actions CI workflow to use msys2/setup-msys2@v2
Main goal is to update the nim-codex CI workflow to use the
[`msys2/setup-msys2@v2`][setmsys2] GitHub Action, as used by the
[nimbus-eth2 workflow][ne2w] per changes made to it several months ago. The
`msys2/setup-msys2@v2` action has been used by other Status-org projects prior
to this commit, e.g. nim-leopard, nim-datastore, nim-status.

A fix is included for the failing macOS builds, related to
[actions/virtual-environments#5819][ave5819]. See [L151][L151].

All builds [succeed][succeed] (with build arch-targets verified as far as
possible via `ldd`, `otool`, `ntldd`), including `linux-i386` and
`windows-i386`, though the `i386` builds are presently disabled (commented
out). The `i386` builds can be enabled simply by uncommenting:

 ```
 # - os: linux
 #   cpu: i386
 ...
 # - os: windows
 #   cpu: i386
 ```

The resulting `.github/workflows/ci.yml` is a "remix" of the current workflows
for nimbus-eth2 and nim-codex (i.e. prior to this commit) along with techniques
learned from developing workflows for other Status-org repos. Some comments and
code-reorg help to clarify/explain what's done in the
`Derive environment variables` step.

`-d:limitStackUsage` has been adopted for `linux-amd64` builds from
[nimbus-eth2's workflow][ne2wL155] and [related code][ne2config] has been
copied into `config.nims`

`-d:limitStackUsage` can easily be dropped if it's not desirable for Codex.

Build targets use `-latest` for `runs-on`, i.e. `macos-latest`,
`ubuntu-latest`, `windows-latest`.

Through a combination of local testing and iterative pushes to GitHub, the
workflow's embedded Bash scripts have been revised to include only the
necessary steps for all builds to succeed, including `linux-i386` and
`windows-i386`.

The GitHub Actions workflow `.github/workflows/codecov.yml` has been removed,
while coverage data generation/upload steps have been added to
`.github/workflows/ci.yml` as the final steps conditional on
`if: runner.os == 'Linux' && matrix.target.cpu == 'amd64' && matrix.nim_branch == matrix.cov_branch`.

A redundant `--passC:'-m32 -mno-adx'` is used for `linux-i386` builds; the
redundant flags do not affect the build, but can be helpful when eyeballing
GitHub Actions builds with increased compile-time verbosity.

Some variable expansions used in `github/workflows/ci.yml` could result in
compilation failures if related paths include whitespace. It's not a problem
for this commit but could be a problem for a user copy-pasting from the
workflow; solving that problem is left as an exercise for the reader.

[setmsys2]: https://github.com/msys2/setup-msys2#readme
[ne2w]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml
[ave5819]: https://github.com/actions/virtual-environments/pull/5819
[L151]: https://github.com/status-im/nim-codex/blob/ci/msys2/.github/workflows/ci.yml#L151
[succeed]: https://github.com/status-im/nim-codex/actions/runs/2606854455
[ne2wL155]: https://github.com/status-im/nimbus-eth2/blob/stable/.github/workflows/ci.yml#L155-L159
[ne2config]: https://github.com/status-im/nimbus-eth2/blob/stable/config.nims#L43-L49
2022-07-06 19:03:10 -05:00
Michael Bradley, Jr 8f0bc54fba test: make `proc finish` async in "nodes negotiate ..." integration test
`waitFor` has been moved into the main body of integration test
`nodes negotiate ...` to reduce blockage on the main thread. That change, along
with widening the expiry in `when defined(windows)` (and bumping up
`timeout-minutes` in our CI workflow) allows Windows CI builds to succeed more
consistently, i.e. without timing out.
2022-07-06 19:03:10 -05:00
Ivan Yonchovski 19407622e7
Merge pull request #126 from yyoncho/bump-libs
Bump libbacktrace and secp256k1
2022-07-04 16:23:27 +03:00
Michael Bradley, Jr 5b34d578d8 windows: ignore *.exe 2022-06-30 17:26:24 -05:00
Michael Bradley, Jr 41bd968ac0 windows: workaround to prevent random segfaults 2022-06-30 17:26:24 -05:00
Bulat-Ziganshin 15ae157e61
BlockStore: delBlock got new type (#123)
Now every store traces only its own operations and failures (currently implemented only for delBlock)
2022-06-28 19:10:05 +03:00
Ivan Yonchovski 9c2224e306 Bump libbacktrace and secp256k1 2022-06-28 11:04:18 +03:00
Dmitriy Ryajov c0b385176d
first pass over the readme (#116)
* first pass over the readme

* wording&typos

* more wording

* add multiaddr link and wording fixes

* add `make update` to instructions

* fix upload command

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* Update README.md

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>

* fix spacing

* adding review suggestion

* more review suggestions

* adding discord badge

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* Update README.md

Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>

* fix levels

Co-authored-by: Eric Mastro <eric.mastro@gmail.com>
Co-authored-by: Michael Bradley <michaelsbradleyjr@gmail.com>
2022-06-23 15:45:42 -06:00
Michael Bradley 2f475115d6
Merge pull request #124 from status-im/build/msys2_fix
build: override CMAKE_ARGS for Windows+MSYS2 environments re: libbacktrace
2022-06-23 16:27:10 -05:00
Michael Bradley, Jr 54616ab727
build: override CMAKE_ARGS for Windows+MSYS2 environments re: libbacktrace
Otherwise `-G"MinGW Makefiles"` is included in `CMAKE_ARGS`, which does not
function correctly in MSYS2 Bash.
2022-06-23 14:11:13 -05:00
Ivan Yonchovski 1088e43c2d
Fix command line args setup (#120)
- commandLineParams is the correct way to access args
2022-06-22 15:37:05 -06:00
Dmitriy Ryajov cf4a0a393e
Json info (#115)
* make info return a json with an spr

* add an `exec` target to the makefile
2022-06-14 17:34:56 -06:00
Dmitriy Ryajov 28ebb16767
adding AsyncStreamWrapper (#110)
* Adding AsyncStreamWrapper type

* use asyncstreamwrapper to avoid copies

* make size an operator

* export asyncstreamwrapper
2022-06-14 09:19:35 -06:00
Dmitriy Ryajov 77733725bf
fix failing test after nim-ethers bump (#112) 2022-06-09 09:36:59 -06:00
Eric Mastro 3b0e93eab4
[chore] bump nim-ethers to 0.1.7 (#107)
* [chore] bump nim-ethers to 0.1.7

* [chore] make block handler callback async

nim-ethers 0.1.7 makes the subscription callbacks async.
2022-06-02 16:09:53 -06:00
Dmitriy Ryajov 6ce7e23767
Upload authenticators (#108)
* initial implementation of storage proofs upload

* make sure proof verifies with after deserializing

* add por store

* rename por store to stp store

* rename porstore to stpstore

* add support for host discovery to discovery mock

* add tags upload network tests
2022-05-25 20:29:31 -06:00
Dmitriy Ryajov 738738c3c6
Adding storage proofs store (#109)
* add por store

* rename por store to stp store

* add stp tests
2022-05-24 18:42:49 -06:00
Dmitriy Ryajov 56b80d6f6d
Por serialize (#106)
* move por into storage proofs

* use SeekableStream

* adding serialization for por

* remove leftovers

* add empty block support

* add basic por test

* rename block exchange for consistency

* add storageproofstests

* moving timing to storageproofs

* fix imports

* fix imports

* fix imports

* add top level exports

* move delete blocks helper to helpers

* more import/export fixes

* cleanup

* more import fixes

* fix unused warnings

* detect corrupt blocks tests

* add serialization tests

* move init method around

* bump asynctest

* fix CID version

* get rid of warning

* wip: fix CI

* increase CI timeout
2022-05-23 23:24:15 -06:00
Dmitriy Ryajov 6ad7a6bb96
Batched blocks (#105)
* remove sleep

* throttle prefetch

* break loop if conn closed

* logging

* prefetch blocks in batches

* add blocks batch constant
2022-05-20 10:53:34 -06:00
Dmitriy Ryajov d4f3ebc867
reorg engine directory structure (#104) 2022-05-19 16:28:53 -06:00
Michael Bradley 209343087c
Change every dagger to codex (#102) 2022-05-19 13:56:03 -06:00
Mark Spanbroek a3f10f7f20 [ci] disable pull-request comments by codecov 2022-05-19 15:23:35 +02:00
Csaba Kiraly aec9a6eb92 enable build on Mac M1
- On Mac, -march=native is only supported in newest Clang versions,
better disable it for now.

- Bumping Leopard version to one that support arm64

Signed-off-by: Csaba Kiraly <csaba.kiraly@gmail.com>
2022-05-19 11:34:49 +02:00