mirror of
https://github.com/status-im/libp2p-test-plans.git
synced 2025-01-13 16:24:15 +00:00
chore: update readme and update references to Testground (#128)
This commit is contained in:
Prithvi Shahi
GitHub
parent
41777186d0
commit
eeb4425242
158
DESIGN.md
158
DESIGN.md
@ -1,158 +0,0 @@
|
||||
# libp2p testing story
|
||||
|
||||
```
|
||||
Date: 2022-10-18
|
||||
Status: In Progress
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
This document describes our process for testing interoperability & backward compatibility across libp2p implementations.
|
||||
|
||||
**Why:**
|
||||
|
||||
- Interoperability is a shared concern.
|
||||
- There is no single blessed libp2p reference implementation that we use for conformance testing.
|
||||
- No single maintainer (go|rust|js-libp2p or IPDX) will succeed without everyone's involvement.
|
||||
- We want to share a Testing Story with the world that shows we care about quality & interop.
|
||||
- We want to encourage other implementations to join the testing party.
|
||||
|
||||
**Historical Context:**
|
||||
|
||||
- We completed a “PING” interop test with Testground. It is running in the go-libp2p and rust-libp2p CI pipeline.
|
||||
- It means we “proved” that we can write and run interop tests between versions AND implementations.
|
||||
|
||||
# Libp2p Testing Matrix
|
||||
|
||||
*What do we want to test next?*
|
||||
|
||||
| | go-libp2p | rust-libp2p | js-libp2p (node) | js-libp2p (browser) | jvm-libp2p | nim-libp2p |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| Simple PING [#35][issue-35] | ✅ | ✅ | 🍎 | 🔥 | | |
|
||||
| Circuit Relay | | | | | | |
|
||||
| WebTransport Transport | 🔥 | | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | | |
|
||||
| WebRTC Transport | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | | |
|
||||
| NAT Traversal | | | | | | |
|
||||
| Hole Punching (STUN) | | | | | | |
|
||||
| Identify Protocol | | | | | | |
|
||||
| AutoNAT | | | | | | |
|
||||
| DHT | | | | | | |
|
||||
| QUIC | | | | | | |
|
||||
| Benchmarking? | | | | | | |
|
||||
|
||||
**Dependencies**
|
||||
|
||||
- Anything `js-libp2p` related requires the `ping` test to start
|
||||
- Benchmarking must relate to [Remote Runners][remote-runners]
|
||||
- https://github.com/testground/testground/pull/1425
|
||||
- https://github.com/testground/testground/issues/1392
|
||||
|
||||
**Questions**
|
||||
|
||||
- When do we revisit this table to discuss priorities and add new tests?
|
||||
|
||||
**Legend**
|
||||
|
||||
- ✅ Done
|
||||
- 🚚 In Progress
|
||||
- 🔥 Highest Priority
|
||||
- 🍎 Low-hanging fruit
|
||||
- 🧊 Lowest priority
|
||||
|
||||
# How does libp2p test interoperability?
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
|
||||
## Background
|
||||
The approach outlined below is pretty much what happen with the go|rust-libp2p ping tests in 2022Q3.
|
||||
|
||||
libp2p implementations aren't forced to adopt this approach, but it is the approach that has been taken by some of the longer-lived implementations (go, JS, and rust).
|
||||
|
||||
I (@laurent) haven’t had time to look at [libp2p/interop](https://github.com/libp2p/interop/actions/runs/3021456724) yet. Some information may be missing.
|
||||
|
||||
## 202210 Proposal
|
||||
<aside>
|
||||
1️⃣ Before working on a new feature, the libp2p maintainers come together and agree on a description of the new test plan.*
|
||||
|
||||
</aside>
|
||||
|
||||
**Example:**
|
||||
|
||||
- [IPFS Test Story in libp2p/interop](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)
|
||||
|
||||
**Question:**
|
||||
|
||||
- What should be the format for this description?
|
||||
- Can we live with a rough “here is a general idea of what the test should do”, and let the first implementor figure out the details?
|
||||
- Do we need to make these decisions now? (09-09-2022)
|
||||
|
||||
<aside>
|
||||
2️⃣ *The maintainers agree on which implementation will provide the reference test implementation (go, rust, js, or other). This implementation is written for Testground and merged in the `libp2p/test-plan` repository.*
|
||||
|
||||
</aside>
|
||||
|
||||
**Example:**
|
||||
|
||||
- https://github.com/libp2p/test-plans/pull/9 “add an instructional libp2p ping test plan”
|
||||
|
||||
**Why:**
|
||||
|
||||
- During implementation, some decisions might be taken on how coordination works, details of the tests, etc. It will be easier to clear the path from one implementation.
|
||||
|
||||
<aside>
|
||||
3️⃣ Once this implementation is merged, the reference implementation enables the test in their CI. It will be a “simple” test that runs the current branch against the last N implementations.
|
||||
|
||||
</aside>
|
||||
|
||||
**Example:**
|
||||
|
||||
- https://github.com/libp2p/go-libp2p/pull/1625 “ci: run testground:ping plan on pull requests” in go-libp2p
|
||||
|
||||
<aside>
|
||||
4️⃣ Other implementation will provide their version of the test. And enable a similar test in CI
|
||||
|
||||
</aside>
|
||||
|
||||
**Example:**
|
||||
|
||||
- https://github.com/libp2p/test-plans/pull/26 “ping/rust: introduce rust cross-version test”
|
||||
- https://github.com/libp2p/rust-libp2p/pull/2835 “.github: introduce interop tests” in rust-libp2p
|
||||
|
||||
<aside>
|
||||
5️⃣ Once multiple implementations have been provided and are running the test in CI, each project will add a “big” test workflow in their Release Process.
|
||||
This “big test” runs the test between every known implementation & version.
|
||||
It might be enabled in a nightly job too.
|
||||
|
||||
</aside>
|
||||
|
||||
**Example:**
|
||||
|
||||
- TODO: add the `full` interop test to `go-libp2p` + update their release documentation.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- When do we revisit this scenario to improve and gather feedback?
|
||||
- How do we evaluate progress & success?
|
||||
- When we’re able to use these tests for benchmarking probably.
|
||||
- What’s the plan for the day when everything starts to break?
|
||||
- What’s the plan for the time when we start to crumble under test complexity?
|
||||
- Maintenance
|
||||
- Tests will need updates on new releases, etc.
|
||||
- What are the dependencies between tests?
|
||||
- ex: Does it make sense to test HOLE PUNCHING if you don’t test AUTONAT first?
|
||||
|
||||
## Refs
|
||||
|
||||
- [https://docs.libp2p.io/concepts/protocols/](https://docs.libp2p.io/concepts/protocols/)
|
||||
- libp2p interop in [Interop Repository](https://github.com/libp2p/interop)
|
||||
- [libp2p interop issue](https://github.com/libp2p/interop/issues/70)
|
||||
- [libp2p/interop test plans](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)
|
||||
|
||||
|
||||
[issue-35]: https://github.com/libp2p/test-plans/issues/35
|
||||
[remote-runners]: https://pl-strflt.notion.site/Remote-Runners-c4ad4886c4294fb6a6f8afd9c0c5b73c
|
||||
46
README.md
46
README.md
@ -1,11 +1,21 @@
|
||||
# Testground test plans for libp2p
|
||||
# Interoperabilty and end to end test-plans for libp2p
|
||||
|
||||
[](https://github.com/libp2p/test-plans/actions/runs/4069611331/attempts/1#summary-11048133990)
|
||||
|
||||
[](http://protocol.ai)
|
||||
|
||||
|
||||
This repository contains Testground test plans for libp2p components.
|
||||
This repository contains interoperability and end to end tests for libp2p modules across different implementations and versions.
|
||||
|
||||
## Specs
|
||||
|
||||
Please see our first specification for interoperability tests between transports, multiplexers, and secure channels here: [Interoperability Test Specs](https://github.com/libp2p/test-plans/blob/master/multidim-interop/README.md)
|
||||
|
||||
More specs to come soon!
|
||||
|
||||
## History
|
||||
|
||||
These test-plans historically used Testground. To read why we're now using `docker compose` instead please see: [Why we're moving away from Testground](https://github.com/libp2p/test-plans/issues/103)
|
||||
|
||||
## Roadmap
|
||||
|
||||
@ -14,38 +24,6 @@ Our roadmap for test-plans can be found here: https://github.com/libp2p/test-pla
|
||||
It represents current projects the test-plans maintainers are focused on and provides an estimation of completion targets.
|
||||
It is complementary to those of [go-libp2p](https://github.com/libp2p/go-libp2p/blob/master/ROADMAP.md), [rust-libp2p](https://github.com/libp2p/rust-libp2p/blob/master/ROADMAP.md), [js-libp2p](https://github.com/libp2p/js-libp2p/blob/master/ROADMAP.md), and the [overarching libp2p project roadmap](https://github.com/libp2p/specs/blob/master/ROADMAP.md).
|
||||
|
||||
## How to add a new version to ping/go
|
||||
|
||||
When a new version of libp2p is released, we want to make it permanent in the `ping/go` test folder.
|
||||
|
||||
1. In the `ping/_compositions/go.toml` file,
|
||||
- copy the `[master]` section and turn it into a `[[groups]]` item
|
||||
- update the `[master]` section with the future version
|
||||
2. In the `ping/go` folder,
|
||||
- Add a new compatibility shim in `compat/` if needed, or add your new selector to the latest shim (see `compat/libp2p.v0.17.go` for example).
|
||||
- Create the new mod and sum files (`go.v0.21.mod` for example). Assuming you're updating from `v$A` to `v$B`, a simple way to do this is to:
|
||||
- `cp go.v$A.mod go.v$B.mod; cp go.v$A.sum go.v$B.sum`
|
||||
- `ln -s go.v$B.mod go.mod; ln -s go.v$B.sum go.sum` (you may also use this for local development, these files are ignored by git)
|
||||
- update the `go-libp2p` version, go version, and update the code if needed.
|
||||
- then `go get -tags v$B && go mod tidy`
|
||||
3. Run the test on your machine
|
||||
- Do once, from the test-plans root: import the test-plans with `testground plan import --from ./ --name libp2p`
|
||||
- Run the test with `testground run composition -f ping/_compositions/go-cross-versions.toml --wait`
|
||||
|
||||
## How to add a new version to ping/rust
|
||||
|
||||
When a new version of libp2p is released, we want to make it permanent in the `ping/rust` test folder.
|
||||
|
||||
1. In the `ping/_compositions/rust.toml` file,
|
||||
- Copy the latest `[[groups]]` section and update it's `Id` and `BinaryName` accordingly.
|
||||
2. In the `ping/rust` folder,
|
||||
- `Cargo.toml`: Add the newly released version as a crates.io dependency.
|
||||
- `Cargo.toml`: Update the `Next release` dependency to the latest `master` SHA.
|
||||
- `src/bin`: Add a new binary with the next released version.
|
||||
3. Run the test on your machine
|
||||
- Do once, from the test-plans root: import the test-plans with `testground plan import --from ./ --name libp2p`
|
||||
- Run the test with `testground run composition -f ping/_compositions/rust-cross-versions.toml --wait`
|
||||
|
||||
## License
|
||||
|
||||
Dual-licensed: [MIT](./LICENSE-MIT), [Apache Software License v2](./LICENSE-APACHE), by way of the
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user