vacp2p
/
libp2p-test-plans
mirror of https://github.com/vacp2p/libp2p-test-plans.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
libp2p-test-plans/ROADMAP.md

12 KiB
Raw Blame History

test-plans roadmap Q4’22/Q1’23 

Date: 2022-10-20
Status: Accepted
Notes: Internal test-plans stakeholders have aligned on this roadmap. Please add any feedback or questions in:
https://github.com/libp2p/test-plans/issues/58

Table of Contents

About the Roadmap

Vision

We, the maintainers, are committed to upholding libp2p's shared core tenets and ensuring libp2p implementations are: Secure, Stable, Specified, and Performant.

This roadmap is complementary to those of go-libp2p, rust-libp2p, and js-libp2p.

It aims to encompass the stability and performance tenets of the libp2p team. Projects outlined here are shared priorities of the different implementations.

Sections

This document consists of two sections: Milestones and the Appendix

Milestones is our best educated guess (not a hard commitment) around when we plan to ship the key features. Where possible projects are broken down into discrete sub-projects e.g. project "A" may contain two sub-projects: A.1 and A.2

A project is signified as "complete" once all of it's sub-projects are shipped.

The Appendix section describes a project's high-level motivation, goals, and lists sub-projects.

Each Appendix header is linked to a GitHub Epic. Latest information on progress can be found in the Epics and child issues.

🛣️ Milestones

2022

Early Q4 (October)

Mid Q4 (November)

End of Q4 (December)

2023

Early Q1 (January)

End of Q1 (March)

Up Next

📖 Appendix

Projects are listed in descending priority.

A. Multi-dimensional Testing/Interop Visibility

Why: libp2p supports a variety of transport protocols, muxers, & security protocols across implementations in different languages. Until we actually test them together, we can't guarantee implementation interoperability. We need to ensure and demonstrate that: interoperable features work with each other as expected and we don't introduce degradations that break interoperability in new releases.

Goal: libp2p implementers run tests across permutations of libp2p implementations, versions, and supported transports, muxers, and security protocols. Implementers and users can reference a single website with a dashboard to view the Pass/Fail/Implemented/Not Implemented results of multi-dimensional tests.

This effort depends on Testground Milestone 1

How:

1. User configured interop tests & dashboard

Enable test-plans maintainers to define a configuration (of libp2p impls, versions, transports, expected RTT), run Testground tests per configuration, and retrieve resulting data in a standard format. The test case results can be formatted as a "dashboard" (simple Markdown table of Pass/Fail results.)

2. Interop test plans for all existing/developing libp2p transports

Using tooling from A.1, all features of go-libp2p, rust-libp2p, and js-libp2p that should be interoperable are tested (i.e. transports (TCP, QUIC, WebRTC, WebTransport), multiplexers (mplex, yamux), secure channels (TLS, Noise), etc.) across versions.

Features currently in development across implementations (like WebRTC in go-libp2p and rust-libp2p, or QUIC & TLS in rust-libp2p) are not merged without at least manually running these test suites.

Test suites are run in libp2p/test-plans CI and before releasing a version of go-libp2p, rust-libp2p, and js-libp2p (GitHub workflow added so that these suites run against the master branch on a nightly basis (updating the status check.))

Note:

3. Canonical interop tests & dashboard

A comprehensive and canonical dashboard is generated and hosted in a publicly discoverable site that displays latest test suite results (Pass/Fail/Implemented/Not Implemented/Unimplementable) from a nightly CI run. All permutations of libp2p implementations, versions, and supported transports, muxers, & security protocols should be visible.

An enhancement of A.1 to make it easier for users and implementers to see the full state of libp2p interoperability.

B. Hardening test infrastructure

1. Track test suite stability

libp2p/test-plans maintainers have a straightforward way to track the test suite stability and performance.

  • We can track the status of every test combination stability from the interop project itself
  • We can easily measure the consequence (improvements) of a pull request to the libp2p/interop repository
  • We are alerted when an interop test starts failing on one of our client repositories, and we can dispatch the alert to repo maintainers.

2. Design process for adding new tests

We have an explicit, working, Design Process for adding new tests

  • The design is documented in ./DESIGN.md.
  • The design is followed by the team when we add new features.
  • There is a clear path when it comes to testing new features. This might mean testing multiple masters against each other.

3. Be the home for all interop tests

We have ported the tests from libp2p/interop

  • This repository implement tests connect, dht, pubsub (ref)
  • At of writing (2022-09-27), it is disabled in go-libp2p (ref), and it is used in js-libp2p (ref).

C. Future-proof Benchmarking

Why: For libp2p to be competitive, it needs to delivers comparable performance to widely used protocols on the internet, namely HTTP/2 and HTTP/3.

Goal: We have a test suite that runs libp2p transfers between nodes located at different locations all over the world, proving that libp2p is able to achieve performance on par with HTTP. The test suite is run on a continuous basis and results are published to a public performance dashboard.

1. Benchmarking using nix-builders

2. Benchmarking using remote runners

Benchmarking using first class support for remote runners (using remote:exec) in Testground

D. Expansive protocol test coverage

Why: Having interoperability tests with lots of transports, encryption mechanisms, and stream muxers is great. However, we need to stay backwards-compatible with legacy libp2p releases, with other libp2p implementations, and less advanced libp2p stacks.

Goal: Expand beyond unit tests and have expansive test-plan coverage that covers all protocols.

This effort depends on Testground Milestone 6

1. DHT server mode scale test

Test js-libp2p DHT Server Mode at scale (testbed of at least >20 nodes; ideally 100/1000+) in Testground Depends on C.1 Relates to Testground Milestone 4 (for large scale tests.)

2. AutoNat

Depends on testground/NAT and/or firewall support

3. Hole Punching

Depends on testground/NAT and/or firewall support

4. AutoRelay

5. Custom topologies

6. MTU Fixes

Depends on testground/Network Simulation Fixes