mirror of https://github.com/status-im/libp2p-test-plans.git synced 2025-02-21 11:08:07 +00:00

History

fix(perf): use $Default launch template (#257 )

Our perf terraform setup differentiates in long lived and short lived resources.
On CI, our long lived resources are spun up once and our short lived resources
are spun up on each CI run. From time to time we have to adjust the long lived
resources. End result is a new launch template that needs to be referenced in
the short lived resources by version.

Next to our CI, the perf terraform setup can as well be used on personal AWS
accounts. Their long lived launch template version likely doesn't match the
configured launch template version of the short lived aws_instance.

Instead of specifying a specific version, instruct terraform to use the default,
thus supporting both our CI and personal AWS account use-case.

2023-08-17 15:02:23 +02:00

impl

fix(perf): replace rust-libp2p-quinn/ with rust-libp2p/master (#246 )

2023-08-01 16:40:02 +02:00

runner

fix(perf): increase maximum udp send buffer size (#254 )

2023-08-14 16:08:41 +02:00

terraform

fix(perf): use $Default launch template (#257 )

2023-08-17 15:02:23 +02:00

Makefile

feat(perf): add (automation, provision, build, run) tooling (#184 )

2023-06-22 13:54:09 +02:00

README.md

docs(perf): document how to add a new version of existing impl (#243 )

2023-07-26 18:41:49 +02:00

README.md

libp2p performance benchmarking

This project includes the following components:

terraform/: a Terraform scripts to provision infrastructure
impl/: implementations of the libp2p perf protocol running on top of e.g. go-libp2p, rust-libp2p or Go's std-library https stack
runner/: a set of scripts building and running the above implementations on the above infrastructure, reporting the results in benchmark-results.json

Benchmark results can be visualized with https://observablehq.com/@libp2p-workspace/performance-dashboard.

Running via GitHub Action

Create a pull request with your changes on https://github.com/libp2p/test-plans/.
Trigger GitHub Action for branch on https://github.com/libp2p/test-plans/actions/workflows/perf.yml (see Run workflow button).
Wait for action run to finish and to push a commit to your branch.
Visualize results on https://observablehq.com/@libp2p-workspace/performance-dashboard.

Running via local machine

Provision infrastructure

Save your public SSH key as the file ./terraform/modules/short_lived/files/perf.pub; or generate a new key pair with make ssh-keygen and add it to your SSH agent with make ssh-add.
cd terraform/configs/local
terraform init
terraform apply
CLIENT_IP=$(terraform output -raw client_ip)
SERVER_IP=$(terraform output -raw server_ip)

Build and run implementations

WARNING: Running the perf tests might take a while.

cd runner
npm ci
npm run start -- --client-public-ip $CLIENT_IP --server-public-ip $SERVER_IP

Deprovision infrastructure

cd terraform/configs/local
terraform destroy

Adding a new implementation or a new version

Add implementation to impl/.
- For a new implementation, create a folder impl/<your-implementation-name>/
- For a new version of an existing implementation, create a folder impl/<your-implementation-name>/<your-implementation-version>.
- In that folder include a Makefile that builds an executable and stores it next to the Makefile under the name perf.
- Requirements for the executable:
  - Running as a libp2p-perf server
    - Command line flags
      - --run-server
  - Running as a libp2p-perf client
    - Input via command line
      - --server-ip-address
      - --transport (see runner/versions.ts for possible variants)
      - --upload-bytes number of bytes to upload per stream.
      - --download-bytes number of bytes to download per stream.
    - Output
      - Logging MUST go to stderr.
      - Measurement output is printed to stdout as JSON in the form of:
        {"latency": 0.246442851}
        Note that the measurement includes the time to (1) establish the connection, (2) upload the bytes and (3) download the bytes.
For a new implementation, in impl/Makefile include your implementation in the all target.
For a new version, reference version in runner/src/versions.ts.