158 lines
7.0 KiB
Markdown
158 lines
7.0 KiB
Markdown
# libp2p testing story
|
||
|
||
```
|
||
Date: 2022-10-18
|
||
Status: In Progress
|
||
```
|
||
|
||
---
|
||
|
||
## Overview
|
||
|
||
This document describes our process for testing interoperability & backward compatibility across libp2p implementations.
|
||
|
||
**Why:**
|
||
|
||
- Interoperability is a shared concern.
|
||
- There is no single blessed libp2p reference implementation that we use for conformance testing.
|
||
- No single maintainer (go|rust|js-libp2p or IPDX) will succeed without everyone's involvement.
|
||
- We want to share a Testing Story with the world that shows we care about quality & interop.
|
||
- We want to encourage other implementations to join the testing party.
|
||
|
||
**Historical Context:**
|
||
|
||
- We completed a “PING” interop test with Testground. It is running in the go-libp2p and rust-libp2p CI pipeline.
|
||
- It means we “proved” that we can write and run interop tests between versions AND implementations.
|
||
|
||
# Libp2p Testing Matrix
|
||
|
||
*What do we want to test next?*
|
||
|
||
| | go-libp2p | rust-libp2p | js-libp2p (node) | js-libp2p (browser) | jvm-libp2p | nim-libp2p |
|
||
| --- | --- | --- | --- | --- | --- | --- |
|
||
| Simple PING [#35][issue-35] | ✅ | ✅ | 🍎 | 🔥 | | |
|
||
| Circuit Relay | | | | | | |
|
||
| WebTransport Transport | 🔥 | | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1) | | |
|
||
| WebRTC Transport | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | 🔥 (depends on working implementation) | | |
|
||
| NAT Traversal | | | | | | |
|
||
| Hole Punching (STUN) | | | | | | |
|
||
| Identify Protocol | | | | | | |
|
||
| AutoNAT | | | | | | |
|
||
| DHT | | | | | | |
|
||
| QUIC | | | | | | |
|
||
| Benchmarking? | | | | | | |
|
||
|
||
**Dependencies**
|
||
|
||
- Anything `js-libp2p` related requires the `ping` test to start
|
||
- Benchmarking must relate to [Remote Runners][remote-runners]
|
||
- https://github.com/testground/testground/pull/1425
|
||
- https://github.com/testground/testground/issues/1392
|
||
|
||
**Questions**
|
||
|
||
- When do we revisit this table to discuss priorities and add new tests?
|
||
|
||
**Legend**
|
||
|
||
- ✅ Done
|
||
- 🚚 In Progress
|
||
- 🔥 Highest Priority
|
||
- 🍎 Low-hanging fruit
|
||
- 🧊 Lowest priority
|
||
|
||
# How does libp2p test interoperability?
|
||
|
||
---
|
||
|
||
---
|
||
|
||
|
||
## Background
|
||
The approach outlined below is pretty much what happen with the go|rust-libp2p ping tests in 2022Q3.
|
||
|
||
libp2p implementations aren't forced to adopt this approach, but it is the approach that has been taken by some of the longer-lived implementations (go, JS, and rust).
|
||
|
||
I (@laurent) haven’t had time to look at [libp2p/interop](https://github.com/libp2p/interop/actions/runs/3021456724) yet. Some information may be missing.
|
||
|
||
## 202210 Proposal
|
||
<aside>
|
||
1️⃣ Before working on a new feature, the libp2p maintainers come together and agree on a description of the new test plan.*
|
||
|
||
</aside>
|
||
|
||
**Example:**
|
||
|
||
- [IPFS Test Story in libp2p/interop](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)
|
||
|
||
**Question:**
|
||
|
||
- What should be the format for this description?
|
||
- Can we live with a rough “here is a general idea of what the test should do”, and let the first implementor figure out the details?
|
||
- Do we need to make these decisions now? (09-09-2022)
|
||
|
||
<aside>
|
||
2️⃣ *The maintainers agree on which implementation will provide the reference test implementation (go, rust, js, or other). This implementation is written for Testground and merged in the `libp2p/test-plan` repository.*
|
||
|
||
</aside>
|
||
|
||
**Example:**
|
||
|
||
- https://github.com/libp2p/test-plans/pull/9 “add an instructional libp2p ping test plan”
|
||
|
||
**Why:**
|
||
|
||
- During implementation, some decisions might be taken on how coordination works, details of the tests, etc. It will be easier to clear the path from one implementation.
|
||
|
||
<aside>
|
||
3️⃣ Once this implementation is merged, the reference implementation enables the test in their CI. It will be a “simple” test that runs the current branch against the last N implementations.
|
||
|
||
</aside>
|
||
|
||
**Example:**
|
||
|
||
- https://github.com/libp2p/go-libp2p/pull/1625 “ci: run testground:ping plan on pull requests” in go-libp2p
|
||
|
||
<aside>
|
||
4️⃣ Other implementation will provide their version of the test. And enable a similar test in CI
|
||
|
||
</aside>
|
||
|
||
**Example:**
|
||
|
||
- https://github.com/libp2p/test-plans/pull/26 “ping/rust: introduce rust cross-version test”
|
||
- https://github.com/libp2p/rust-libp2p/pull/2835 “.github: introduce interop tests” in rust-libp2p
|
||
|
||
<aside>
|
||
5️⃣ Once multiple implementations have been provided and are running the test in CI, each project will add a “big” test workflow in their Release Process.
|
||
This “big test” runs the test between every known implementation & version.
|
||
It might be enabled in a nightly job too.
|
||
|
||
</aside>
|
||
|
||
**Example:**
|
||
|
||
- TODO: add the `full` interop test to `go-libp2p` + update their release documentation.
|
||
|
||
## Open Questions
|
||
|
||
- When do we revisit this scenario to improve and gather feedback?
|
||
- How do we evaluate progress & success?
|
||
- When we’re able to use these tests for benchmarking probably.
|
||
- What’s the plan for the day when everything starts to break?
|
||
- What’s the plan for the time when we start to crumble under test complexity?
|
||
- Maintenance
|
||
- Tests will need updates on new releases, etc.
|
||
- What are the dependencies between tests?
|
||
- ex: Does it make sense to test HOLE PUNCHING if you don’t test AUTONAT first?
|
||
|
||
## Refs
|
||
|
||
- [https://docs.libp2p.io/concepts/protocols/](https://docs.libp2p.io/concepts/protocols/)
|
||
- libp2p interop in [Interop Repository](https://github.com/libp2p/interop)
|
||
- [libp2p interop issue](https://github.com/libp2p/interop/issues/70)
|
||
- [libp2p/interop test plans](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)
|
||
|
||
|
||
[issue-35]: https://github.com/libp2p/test-plans/issues/35
|
||
[remote-runners]: https://pl-strflt.notion.site/Remote-Runners-c4ad4886c4294fb6a6f8afd9c0c5b73c |