status-im/libp2p-test-plans
2
0
Fork 0
mirror of https://github.com/status-im/libp2p-test-plans.git synced 2025-01-13 08:14:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: update readme and update references to Testground (#128)

Browse Source
This commit is contained in:
Prithvi Shahi 2023-02-07 12:10:24 -08:00 committed by GitHub
parent 41777186d0
commit eeb4425242
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 12 additions and 192 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

158
DESIGN.md
View File

@ -1,158 +0,0 @@
# libp2p testing story
```
Date: 2022-10-18
Status: In Progress
```
---
## Overview
This document describes our process for testing interoperability & backward compatibility across libp2p implementations.
**Why:**
- Interoperability is a shared concern.
- There is no single blessed libp2p reference implementation that we use for conformance testing.
- No single maintainer (go|rust|js-libp2p or IPDX) will succeed without everyone's involvement.
- We want to share a Testing Story with the world that shows we care about quality & interop.
- We want to encourage other implementations to join the testing party.
**Historical Context:**
- We completed a “PING” interop test with Testground. It is running in the go-libp2p and rust-libp2p CI pipeline.
- It means we “proved” that we can write and run interop tests between versions AND implementations.
# Libp2p Testing Matrix
*What do we want to test next?*
| | go-libp2p | rust-libp2p | js-libp2p (node) | js-libp2p (browser) | jvm-libp2p | nim-libp2p |
| --- | --- | --- | --- | --- | --- | --- |
| Simple PING [#35][issue-35] | ✅ | ✅ | 🍎 | 🔥 | | |
| Circuit Relay | | | | | | |
| WebTransport Transport | 🔥 | | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | | |
| WebRTC Transport | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | | |
| NAT Traversal | | | | | | |
| Hole Punching (STUN) | | | | | | |
| Identify Protocol | | | | | | |
| AutoNAT | | | | | | |
| DHT | | | | | | |
| QUIC | | | | | | |
| Benchmarking? | | | | | | |
**Dependencies**
- Anything `js-libp2p` related requires the `ping` test to start
- Benchmarking must relate to [Remote Runners][remote-runners]
- https://github.com/testground/testground/pull/1425
- https://github.com/testground/testground/issues/1392
**Questions**
- When do we revisit this table to discuss priorities and add new tests?
**Legend**
- ✅ Done
- 🚚 In Progress
- 🔥 Highest Priority
- 🍎 Low-hanging fruit
- 🧊 Lowest priority
# How does libp2p test interoperability?
---
---
## Background
The approach outlined below is pretty much what happen with the go|rust-libp2p ping tests in 2022Q3.
libp2p implementations aren't forced to adopt this approach, but it is the approach that has been taken by some of the longer-lived implementations (go, JS, and rust).
I (@laurent) haven’t had time to look at [libp2p/interop](https://github.com/libp2p/interop/actions/runs/3021456724) yet. Some information may be missing.
## 202210 Proposal
<aside>
1️⃣ Before working on a new feature, the libp2p maintainers come together and agree on a description of the new test plan.*
</aside>
**Example:**
- [IPFS Test Story in libp2p/interop](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)
**Question:**
- What should be the format for this description?
- Can we live with a rough “here is a general idea of what the test should do”, and let the first implementor figure out the details?
- Do we need to make these decisions now? (09-09-2022)
<aside>
2️⃣ *The maintainers agree on which implementation will provide the reference test implementation (go, rust, js, or other). This implementation is written for Testground and merged in the `libp2p/test-plan` repository.*
</aside>
**Example:**
- https://github.com/libp2p/test-plans/pull/9 “add an instructional libp2p ping test plan”
**Why:**
- During implementation, some decisions might be taken on how coordination works, details of the tests, etc. It will be easier to clear the path from one implementation.
<aside>
3️⃣ Once this implementation is merged, the reference implementation enables the test in their CI. It will be a “simple” test that runs the current branch against the last N implementations.
</aside>
**Example:**
- https://github.com/libp2p/go-libp2p/pull/1625 “ci: run testground:ping plan on pull requests” in go-libp2p
<aside>
4️⃣ Other implementation will provide their version of the test. And enable a similar test in CI
</aside>
**Example:**
- https://github.com/libp2p/test-plans/pull/26 “ping/rust: introduce rust cross-version test”
- https://github.com/libp2p/rust-libp2p/pull/2835 “.github: introduce interop tests” in rust-libp2p
<aside>
5️⃣ Once multiple implementations have been provided and are running the test in CI, each project will add a “big” test workflow in their Release Process.
This “big test” runs the test between every known implementation & version.
It might be enabled in a nightly job too.
</aside>
**Example:**
- TODO: add the `full` interop test to `go-libp2p` + update their release documentation.
## Open Questions
- When do we revisit this scenario to improve and gather feedback?
- How do we evaluate progress & success?
- When we’re able to use these tests for benchmarking probably.
- What’s the plan for the day when everything starts to break?
- What’s the plan for the time when we start to crumble under test complexity?
- Maintenance
- Tests will need updates on new releases, etc.
- What are the dependencies between tests?
- ex: Does it make sense to test HOLE PUNCHING if you don’t test AUTONAT first?
## Refs
- [https://docs.libp2p.io/concepts/protocols/](https://docs.libp2p.io/concepts/protocols/)
- libp2p interop in [Interop Repository](https://github.com/libp2p/interop)
- [libp2p interop issue](https://github.com/libp2p/interop/issues/70)
- [libp2p/interop test plans](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)
[issue-35]: https://github.com/libp2p/test-plans/issues/35
[remote-runners]: https://pl-strflt.notion.site/Remote-Runners-c4ad4886c4294fb6a6f8afd9c0c5b73c

46
README.md
View File

@ -1,11 +1,21 @@
# Testground test plans for libp2p
# Interoperabilty and end to end test-plans for libp2p
[![Interop Dashboard](https://github.com/libp2p/test-plans/workflows/libp2p%20multidimensional%20interop%20test/badge.svg?branch=master)](https://github.com/libp2p/test-plans/actions/runs/4069611331/attempts/1#summary-11048133990)
[![Made by Protocol Labs](https://img.shields.io/badge/made%20by-Protocol%20Labs-blue.svg?style=flat-square)](http://protocol.ai)
![Go version](https://img.shields.io/badge/go-%3E%3D1.14.0-blue.svg?style=flat-square)
This repository contains Testground test plans for libp2p components.
This repository contains interoperability and end to end tests for libp2p modules across different implementations and versions.
## Specs
Please see our first specification for interoperability tests between transports, multiplexers, and secure channels here: [Interoperability Test Specs](https://github.com/libp2p/test-plans/blob/master/multidim-interop/README.md)
More specs to come soon!
## History
These test-plans historically used Testground. To read why we're now using `docker compose` instead please see: [Why we're moving away from Testground](https://github.com/libp2p/test-plans/issues/103)
## Roadmap
@ -14,38 +24,6 @@ Our roadmap for test-plans can be found here: https://github.com/libp2p/test-pla
It represents current projects the test-plans maintainers are focused on and provides an estimation of completion targets.
It is complementary to those of [go-libp2p](https://github.com/libp2p/go-libp2p/blob/master/ROADMAP.md), [rust-libp2p](https://github.com/libp2p/rust-libp2p/blob/master/ROADMAP.md), [js-libp2p](https://github.com/libp2p/js-libp2p/blob/master/ROADMAP.md), and the [overarching libp2p project roadmap](https://github.com/libp2p/specs/blob/master/ROADMAP.md).
## How to add a new version to ping/go
When a new version of libp2p is released, we want to make it permanent in the `ping/go` test folder.
1. In the `ping/_compositions/go.toml` file,
- copy the `[master]` section and turn it into a `[[groups]]` item
- update the `[master]` section with the future version
2. In the `ping/go` folder,
- Add a new compatibility shim in `compat/` if needed, or add your new selector to the latest shim (see `compat/libp2p.v0.17.go` for example).
- Create the new mod and sum files (`go.v0.21.mod` for example). Assuming you're updating from `v$A` to `v$B`, a simple way to do this is to:
- `cp go.v$A.mod go.v$B.mod; cp go.v$A.sum go.v$B.sum`
- `ln -s go.v$B.mod go.mod; ln -s go.v$B.sum go.sum` (you may also use this for local development, these files are ignored by git)
- update the `go-libp2p` version, go version, and update the code if needed.
- then `go get -tags v$B && go mod tidy`
3. Run the test on your machine
- Do once, from the test-plans root: import the test-plans with `testground plan import --from ./ --name libp2p`
- Run the test with `testground run composition -f ping/_compositions/go-cross-versions.toml --wait`
## How to add a new version to ping/rust
When a new version of libp2p is released, we want to make it permanent in the `ping/rust` test folder.
1. In the `ping/_compositions/rust.toml` file,
- Copy the latest `[[groups]]` section and update it's `Id` and `BinaryName` accordingly.
2. In the `ping/rust` folder,
- `Cargo.toml`: Add the newly released version as a crates.io dependency.
- `Cargo.toml`: Update the `Next release` dependency to the latest `master` SHA.
- `src/bin`: Add a new binary with the next released version.
3. Run the test on your machine
- Do once, from the test-plans root: import the test-plans with `testground plan import --from ./ --name libp2p`
- Run the test with `testground run composition -f ping/_compositions/rust-cross-versions.toml --wait`
## License
Dual-licensed: [MIT](./LICENSE-MIT), [Apache Software License v2](./LICENSE-APACHE), by way of the