libp2p-test-plans/DESIGN.md

libp2p testing story

Date: 2022-10-18
Status: In Progress

---

Overview

This document describes our process for testing interoperability & backward compatibility across libp2p implementations.

Why:

• Interoperability is a shared concern.
• There is no single blessed libp2p reference implementation that we use for conformance testing.
• No single maintainer (go|rust|js-libp2p or IPDX) will succeed without everyone's involvement.
• We want to share a Testing Story with the world that shows we care about quality & interop.
• We want to encourage other implementations to join the testing party.

Historical Context:

• We completed a “PING” interop test with Testground. It is running in the go-libp2p and rust-libp2p CI pipeline.
• It means we “proved” that we can write and run interop tests between versions AND implementations.

Libp2p Testing Matrix

What do we want to test next?

	go-libp2p	rust-libp2p	js-libp2p (node)	js-libp2p (browser)	jvm-libp2p	nim-libp2p
Simple PING #35	✅	✅	🍎	🔥
Circuit Relay
WebTransport Transport	🔥		🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1)	🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1)
WebRTC Transport	🔥 (depends on working implementation)	🔥 (depends on working implementation)	🔥 (depends on working implementation)	🔥 (depends on working implementation)
NAT Traversal
Hole Punching (STUN)
Identify Protocol
AutoNAT
DHT
QUIC
Benchmarking?

Dependencies

• Anything js-libp2p related requires the ping test to start
• Benchmarking must relate to Remote Runners
  • https://github.com/testground/testground/pull/1425
  • https://github.com/testground/testground/issues/1392

Questions

• When do we revisit this table to discuss priorities and add new tests?

Legend

• ✅ Done
• 🚚 In Progress
• 🔥 Highest Priority
• 🍎 Low-hanging fruit
• 🧊 Lowest priority

How does libp2p test interoperability?

---

---

Background

The approach outlined below is pretty much what happen with the go|rust-libp2p ping tests in 2022Q3.

libp2p implementations aren't forced to adopt this approach, but it is the approach that has been taken by some of the longer-lived implementations (go, JS, and rust).

I (@laurent) haven’t had time to look at libp2p/interop yet. Some information may be missing.

202210 Proposal

1️⃣ Before working on a new feature, the libp2p maintainers come together and agree on a description of the new test plan.*

Example:

• IPFS Test Story in libp2p/interop

Question:

• What should be the format for this description?
• Can we live with a rough “here is a general idea of what the test should do”, and let the first implementor figure out the details?
• Do we need to make these decisions now? (09-09-2022)

2️⃣ *The maintainers agree on which implementation will provide the reference test implementation (go, rust, js, or other). This implementation is written for Testground and merged in the `libp2p/test-plan` repository.*

Example:

• https://github.com/libp2p/test-plans/pull/9 “add an instructional libp2p ping test plan”

Why:

• During implementation, some decisions might be taken on how coordination works, details of the tests, etc. It will be easier to clear the path from one implementation.

3️⃣ Once this implementation is merged, the reference implementation enables the test in their CI. It will be a “simple” test that runs the current branch against the last N implementations.

Example:

• https://github.com/libp2p/go-libp2p/pull/1625 “ci: run testground:ping plan on pull requests” in go-libp2p

4️⃣ Other implementation will provide their version of the test. And enable a similar test in CI

Example:

• https://github.com/libp2p/test-plans/pull/26 “ping/rust: introduce rust cross-version test”
• https://github.com/libp2p/rust-libp2p/pull/2835 “.github: introduce interop tests” in rust-libp2p

5️⃣ Once multiple implementations have been provided and are running the test in CI, each project will add a “big” test workflow in their Release Process. This “big test” runs the test between every known implementation & version. It might be enabled in a nightly job too.

Example:

• TODO: add the full interop test to go-libp2p + update their release documentation.

Open Questions

• When do we revisit this scenario to improve and gather feedback?
  • How do we evaluate progress & success?
    • When we’re able to use these tests for benchmarking probably.
  • What’s the plan for the day when everything starts to break?
  • What’s the plan for the time when we start to crumble under test complexity?
• Maintenance
  • Tests will need updates on new releases, etc.
• What are the dependencies between tests?
  • ex: Does it make sense to test HOLE PUNCHING if you don’t test AUTONAT first?

Refs

• https://docs.libp2p.io/concepts/protocols/
• libp2p interop in Interop Repository
• libp2p interop issue
• libp2p/interop test plans