# libp2p testing story

```
Date: 2022-10-18
Status: In Progress
```

---

## Overview

This document describes our process for testing interoperability & backward compatibility across libp2p implementations.

**Why:**

- Interoperability is a shared concern.
- There is no single blessed libp2p reference implementation that we use for conformance testing.
- No single maintainer (go|rust|js-libp2p or IPDX) will succeed without everyone's involvement.
- We want to share a Testing Story with the world that shows we care about quality & interop.
- We want to encourage other implementations to join the testing party.

**Historical Context:**

- We completed a “PING” interop test with Testground. It is running in the go-libp2p and rust-libp2p CI pipeline.
- It means we “proved” that we can write and run interop tests between versions AND implementations.

# Libp2p Testing Matrix

*What do we want to test next?*

|                                   | go-libp2p | rust-libp2p | js-libp2p (node) | js-libp2p (browser) | jvm-libp2p | nim-libp2p |
| ---                               | ---       | ---         | ---              | ---                 | ---        | ---        |
| Simple PING [#35][issue-35]       | ✅        | ✅          | 🍎               | 🔥                  |            |            |
| Circuit Relay                     |           |             |                  |                     |            |            |
| WebTransport Transport            | 🔥        |           | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1)               | 🔥 (depends on https://github.com/libp2p/js-libp2p-webtransport/issues/1)                  |          |          |
| WebRTC Transport                  | 🔥 (depends on working implementation)        | 🔥 (depends on working implementation)          | 🔥 (depends on working implementation)               | 🔥 (depends on working implementation)                  |          |          |
| NAT Traversal                     |           |             |                  |                     |            |            |
| Hole Punching (STUN)              |           |             |                  |                     |            |            |
| Identify Protocol                 |           |             |                  |                     |            |            |
| AutoNAT                           |           |             |                  |                     |            |            |
| DHT                               |           |             |                  |                     |            |            |
| QUIC                              |           |             |                  |                     |            |            |
| Benchmarking?                     |           |             |                  |                     |            |            |

**Dependencies**

- Anything `js-libp2p` related requires the `ping` test to start
- Benchmarking must relate to [Remote Runners][remote-runners]
  - https://github.com/testground/testground/pull/1425
  - https://github.com/testground/testground/issues/1392

**Questions**

- When do we revisit this table to discuss priorities and add new tests?

**Legend**

- ✅ Done
- 🚚 In Progress
- 🔥 Highest Priority
- 🍎 Low-hanging fruit
- 🧊 Lowest priority

# How does libp2p test interoperability?

---

---


## Background
The approach outlined below is pretty much what happen with the go|rust-libp2p ping tests in 2022Q3.

libp2p implementations aren't forced to adopt this approach, but it is the approach that has been taken by some of the longer-lived implementations (go, JS, and rust).  

I (@laurent) haven’t had time to look at [libp2p/interop](https://github.com/libp2p/interop/actions/runs/3021456724) yet. Some information may be missing.

## 202210 Proposal
<aside>
1️⃣ Before working on a new feature, the libp2p maintainers come together and agree on a description of the new test plan.*

</aside>

**Example:**

- [IPFS Test Story in libp2p/interop](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)

**Question:**

- What should be the format for this description?
- Can we live with a rough “here is a general idea of what the test should do”, and let the first implementor figure out the details?
- Do we need to make these decisions now? (09-09-2022)

<aside>
2️⃣ *The maintainers agree on which implementation will provide the reference test implementation (go, rust, js, or other). This implementation is written for Testground and merged in the `libp2p/test-plan` repository.*

</aside>

**Example:**

- https://github.com/libp2p/test-plans/pull/9 “add an instructional libp2p ping test plan”

**Why:**

- During implementation, some decisions might be taken on how coordination works, details of the tests, etc. It will be easier to clear the path from one implementation.

<aside>
3️⃣ Once this implementation is merged, the reference implementation enables the test in their CI. It will be a “simple” test that runs the current branch against the last N implementations.

</aside>

**Example:**

- https://github.com/libp2p/go-libp2p/pull/1625 “ci: run testground:ping plan on pull requests” in go-libp2p

<aside>
4️⃣ Other implementation will provide their version of the test. And enable a similar test in CI

</aside>

**Example:**

- https://github.com/libp2p/test-plans/pull/26 “ping/rust: introduce rust cross-version test”
- https://github.com/libp2p/rust-libp2p/pull/2835 “.github: introduce interop tests” in rust-libp2p

<aside>
5️⃣ Once multiple implementations have been provided and are running the test in CI, each project will add a “big” test workflow in their Release Process.
This “big test” runs the test between every known implementation & version.
It might be enabled in a nightly job too.

</aside>

**Example:**

- TODO: add the `full` interop test to `go-libp2p` + update their release documentation.

## Open Questions

- When do we revisit this scenario to improve and gather feedback?
    - How do we evaluate progress & success?
        - When we’re able to use these tests for benchmarking probably.
    - What’s the plan for the day when everything starts to break?
    - What’s the plan for the time when we start to crumble under test complexity?
- Maintenance
    - Tests will need updates on new releases, etc.
- What are the dependencies between tests?
    - ex: Does it make sense to test HOLE PUNCHING if you don’t test AUTONAT first?

## Refs

- [https://docs.libp2p.io/concepts/protocols/](https://docs.libp2p.io/concepts/protocols/)
- libp2p interop in [Interop Repository](https://github.com/libp2p/interop)
- [libp2p interop issue](https://github.com/libp2p/interop/issues/70)
- [libp2p/interop test plans](https://github.com/libp2p/interop/blob/master/pdd/PDD-THE-IPFS-BUNDLE.md)


[issue-35]: https://github.com/libp2p/test-plans/issues/35
[remote-runners]: https://pl-strflt.notion.site/Remote-Runners-c4ad4886c4294fb6a6f8afd9c0c5b73c