logos-messaging/js-waku
1
0
Fork 0
mirror of https://github.com/logos-messaging/js-waku.git synced 2026-01-08 08:43:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: specify lightpush peers for browser sim using cli

Browse Source
This commit is contained in:
Arseniy Klempner 2025-09-03 22:34:58 -07:00
parent e901f24c9d
commit f432d6f42d
No known key found for this signature in database
GPG Key ID: 51653F18863BD24B
6 changed files with 262 additions and 148 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
PR.md
View File

@ -1,27 +0,0 @@
### Problem / Description
- Duplicate browser testing packages increased maintenance and broke CI (referenced removed `@waku/headless-tests`).
- Dockerized tests failed in CI runners without Docker access.
### Solution
- Consolidated all browser/headless tests into a single package: `@waku/browser-tests` (removed `packages/headless-tests`).
- Introduced lightweight bootstrap (`src/assets/bootstrap.js`) and `shared/` module; simplified routes and server.
- Replaced root-level Dockerfile with a package-local Dockerfile under `packages/browser-tests`.
- Build image: `cd packages/browser-tests && npm run docker:build`
- Run dockerized tests: `HEADLESS_USE_CDN_IN_DOCKER=0 npx playwright test tests/docker-server.spec.ts`
- Fixed Playwright CI to build/test `@waku/browser-tests`; skip Docker-based tests on CI via Playwright `testIgnore`.
### Notes
- Docker tests require a Docker-enabled environment and local image build; they are intentionally skipped in CI.
- Resolves: CI failures from removed workspace and duplicated setup.
- Related to: test infra consolidation and stability.
---
#### Checklist
- [ ] Code changes are **covered by unit tests**.
- [ ] Code changes are **covered by e2e tests**, if applicable.
- [ ] **Dogfooding has been performed**, if feasible.
- [ ] A **test version has been published**, if required.
- [ ] All **CI checks** pass successfully.

2
packages/browser-tests/.dockerignore
View File

@ -2,5 +2,3 @@ node_modules
build
.DS_Store
*.log
# Don't ignore dist - we need the built files
# dist

220
packages/browser-tests/README.md
View File

@ -1,178 +1,182 @@
# Waku Browser Tests
Browser-simulated js-waku node running inside headless Chromium, controlled by an Express server. Useful for long-running simulations and realistic verification in CI/Docker.
This project provides a system for testing the Waku SDK in a browser environment.
## Architecture
- **Headless browser**: Playwright launches Chromium and navigates to a static site built from TypeScript and served by the same server, which exposes `window.wakuAPI` and `window.waku`.
- **Server**: Express app provides REST endpoints and proxies calls into the browser via `page.evaluate(...)`.
- **Bootstrap module**: Small browser-side module in the static site initializes the API and imports `@waku/sdk`.
- **Shared code**: `shared/` contains utilities used by tests and for typing.
The system consists of:
## Prerequisites
1. **Headless Web App**: A simple web application (in the `@waku/headless-tests` package) that loads the Waku SDK and exposes shared API functions.
2. **Express Server**: A server that communicates with the headless app using Playwright.
3. **Shared API**: TypeScript functions shared between the server and web app.
- Node.js 18+
- Playwright (installed via dev dependency)
- Docker (optional, for Testcontainers-based tests)
## Setup
## Install & Build
1. Install dependencies:
```bash
# Install main dependencies
npm install
# Install headless app dependencies
cd ../headless-tests
npm install
cd ../browser-tests
```
2. Build the application:
```bash
npm run build
```
The build compiles the TypeScript server to `dist/` and bundles the static site to `dist/web/`.
This will:
- Build the headless web app using webpack
- Compile the TypeScript server code
## Run
## Running
Start the server with:
```bash
# Default configuration (cluster ID 1, auto-sharding)
npm run start:server
# Use cluster ID 2 (for 10k sim compatibility)
npm run start:cluster2
# Use specific cluster and shard
npm run start:cluster2-shard0
# Or with direct CLI arguments
npm run build && node dist/src/server.js --cluster-id=2 --shard=3
```
This starts the API server and a headless browser.
### CLI Arguments
- `--cluster-id=N` - Set the Waku cluster ID (default: 1)
- `--shard=N` - Set a specific shard for static sharding (0-7, omit for auto-sharding)
## Environment variables
- `PORT`: API server port (default: 8080; Playwright sets this for tests)
- `WAKU_WS_MULTIADDR`: a single ws/wss multiaddr to dial in tests (overrides peers)
- `WAKU_WS_MULTIADDRS`: multiple peers as JSON array (e.g. `["/dns4/.../wss/p2p/16U..."]`) or comma-separated string; used when `WAKU_WS_MULTIADDR` is not set
This will:
1. Serve the headless app on port 8080
2. Start a headless browser to load the app
3. Expose API endpoints to interact with Waku
## API Endpoints
- `GET /` – health/status
- `GET /info` – peer info from the node
- `GET /debug/v1/info` – debug info/protocols
- `POST /lightpush/v1/message` – push a message (Waku REST-compatible shape)
- `POST /admin/v1/create-node` – create a node with `networkConfig`
- `POST /admin/v1/start-node` – start the node
- `POST /admin/v1/stop-node` – stop the node
- `POST /admin/v1/peers` – dial to peers
- `GET /filter/v2/messages/:contentTopic` – SSE subscription to messages
- `GET /filter/v1/messages/:contentTopic` – retrieve queued messages
- `POST /execute` – helper to execute functions in the browser context (testing/support)
- `GET /info`: Get information about the Waku node
- `GET /debug/v1/info`: Get debug information from the Waku node
- `POST /push`: Push a message to the Waku network (legacy)
- `POST /lightpush/v1/message`: Push a message to the Waku network (Waku REST API compatible)
- `POST /admin/v1/create-node`: Create a new Waku node (requires networkConfig)
- `POST /admin/v1/start-node`: Start the Waku node
- `POST /admin/v1/stop-node`: Stop the Waku node
- `POST /admin/v1/peers`: Dial to specified peers (Waku REST API compatible)
- `GET /filter/v2/messages/:contentTopic`: Subscribe to messages on a specific content topic using Server-Sent Events (Waku REST API compatible)
- `GET /filter/v1/messages/:contentTopic`: Retrieve stored messages from a content topic (Waku REST API compatible)
### Examples
### Example: Pushing a message with the legacy endpoint
Push (REST-compatible):
```bash
curl -X POST http://localhost:3000/push \
-H "Content-Type: application/json" \
-d '{"contentTopic": "/toy-chat/2/huilong/proto", "payload": [1, 2, 3]}'
```
### Example: Pushing a message with the Waku REST API compatible endpoint
```bash
curl -X POST http://localhost:3000/lightpush/v1/message \
-H "Content-Type: application/json" \
-d '{
"pubsubTopic": "/waku/2/rs/1/0",
"pubsubTopic": "/waku/2/rs/0/0",
"message": {
"payload": [1,2,3],
"contentTopic": "/test/1/message/proto"
"payload": "SGVsbG8sIFdha3Uh",
"contentTopic": "/toy-chat/2/huilong/proto",
"timestamp": 1712135330213797632
}
}'
```
Create/Start/Stop:
### Example: Executing a function
```bash
curl -X POST http://localhost:3000/execute \
-H "Content-Type: application/json" \
-d '{"functionName": "getPeerInfo", "params": []}'
```
### Example: Creating a Waku node
```bash
curl -X POST http://localhost:3000/admin/v1/create-node \
-H "Content-Type: application/json" \
-d '{
"defaultBootstrap": true,
"networkConfig": { "clusterId": 42, "shards": [0] }
"networkConfig": {
"clusterId": 1,
"shards": [0, 1]
}
}'
```
### Example: Starting and stopping a Waku node
```bash
# Start the node
curl -X POST http://localhost:3000/admin/v1/start-node
# Stop the node
curl -X POST http://localhost:3000/admin/v1/stop-node
```
Dial peers:
### Example: Dialing to specific peers with the Waku REST API compatible endpoint
```bash
curl -X POST http://localhost:3000/admin/v1/peers \
-H "Content-Type: application/json" \
-d '{
"peerMultiaddrs": ["/dns4/example/tcp/8000/wss/p2p/16U..."]
"peerMultiaddrs": [
"/ip4/127.0.0.1/tcp/8000/p2p/16Uiu2HAm4v8KuHUH6Cwz3upPeQbkyxQJsFGPdt7kHtkN8F79QiE6"]
]
}'
```
SSE subscribe:
### Example: Dialing to specific peers with the execute endpoint
```bash
curl -N "http://localhost:3000/filter/v2/messages/test-topic?clusterId=1&shard=0"
curl -X POST http://localhost:3000/execute \
-H "Content-Type: application/json" \
-d '{
"functionName": "dialPeers",
"params": [
["/ip4/127.0.0.1/tcp/8000/p2p/16Uiu2HAm4v8KuHUH6Cwz3upPeQbkyxQJsFGPdt7kHtkN8F79QiE6"]
]
}'
```
Query queued messages:
### Example: Subscribing to a content topic with the filter endpoint
```bash
curl "http://localhost:3000/filter/v1/messages/test-topic?pageSize=10&ascending=true"
# Open a persistent connection to receive messages as Server-Sent Events
curl -N http://localhost:3000/filter/v2/messages/%2Ftoy-chat%2F2%2Fhuilong%2Fproto
# You can also specify clustering options
curl -N "http://localhost:3000/filter/v2/messages/%2Ftoy-chat%2F2%2Fhuilong%2Fproto?clusterId=0&shard=0"
```
## Testing
### Example: Retrieving stored messages from a content topic
```bash
npm run build
npm test
# Get the most recent 20 messages
curl http://localhost:3000/filter/v1/messages/%2Ftoy-chat%2F2%2Fhuilong%2Fproto
# Get messages with pagination and time filtering
curl "http://localhost:3000/filter/v1/messages/%2Ftoy-chat%2F2%2Fhuilong%2Fproto?pageSize=10&startTime=1712000000000&endTime=1713000000000&ascending=true"
```
Playwright will start the server (uses `npm run start:server`). Ensure the build artifacts exist before running tests.
## Docker Usage
Build and run with default configuration:
```bash
npm run docker:build
docker run -p 8080:8080 waku-browser-tests:local
```
Run with cluster ID 2 for 10k sim compatibility:
```bash
docker run -p 8080:8080 waku-browser-tests:local --cluster-id=2
```
Run with specific cluster and shard:
```bash
docker run -p 8080:8080 waku-browser-tests:local --cluster-id=2 --shard=0
```
Or using environment variables:
```bash
docker run -p 8080:8080 -e WAKU_CLUSTER_ID=2 -e WAKU_SHARD=0 waku-browser-tests:local
```
### Dockerized tests
`tests/docker-server.spec.ts` uses Testcontainers. Ensure Docker is running.
Build the image and run only the docker tests locally:
```bash
npm run docker:build
npx playwright test tests/docker-server.spec.ts
```
Notes:
- The Docker image runs the server with Playwright Chromium and `--no-sandbox` for container compatibility.
- Testcontainers will map the container port automatically; the tests probe readiness by waiting for `API server running on http://localhost:` in logs.
## Extending
- To add new REST endpoints: update `src/server.ts` and route handlers.
- To add new browser-executed functions: prefer updating `src/assets/bootstrap.js` (minimize inline JS in `src/server.ts`).
- For shared logic usable in tests, add helpers under `shared/`.
To add new functionality:
1. Add your function to `src/api/shared.ts`
2. Add your function to the `API` object in `src/api/shared.ts`
3. Use it via the server endpoints
### Example: Dialing to specific peers
```bash
curl -X POST http://localhost:3000/execute \
-H "Content-Type: application/json" \
-d '{
"functionName": "dialPeers",
"params": [
["/ip4/127.0.0.1/tcp/8000/p2p/16Uiu2HAm4v8KuHUH6Cwz3upPeQbkyxQJsFGPdt7kHtkN8F79QiE6"]
]
}'
```

16
packages/browser-tests/scripts/docker-entrypoint.sh
View File

@ -2,6 +2,17 @@
# Docker entrypoint script for waku-browser-tests
# Handles CLI arguments and converts them to environment variables
# Supports reading discovered addresses from /etc/addrs/addrs.env (10k sim pattern)
# Check if address file exists and source it
if [ -f "/etc/addrs/addrs.env" ]; then
echo "Sourcing discovered addresses from /etc/addrs/addrs.env"
source /etc/addrs/addrs.env
if [ -n "$addrs1" ]; then
export WAKU_LIGHTPUSH_NODE="$addrs1"
echo "Using discovered lightpush node: $WAKU_LIGHTPUSH_NODE"
fi
fi
# Parse command line arguments
while [[ $# -gt 0 ]]; do
@ -16,6 +27,11 @@ while [[ $# -gt 0 ]]; do
echo "Setting WAKU_SHARD=${WAKU_SHARD}"
shift
;;
--lightpushnode=*)
export WAKU_LIGHTPUSH_NODE="${1#*=}"
echo "Setting WAKU_LIGHTPUSH_NODE=${WAKU_LIGHTPUSH_NODE}"
shift
;;
*)
# Unknown argument, keep it for the main command
break

25
packages/browser-tests/src/server.ts
View File

@ -40,8 +40,14 @@ app.get("/app/index.html", (_req: Request, res: Response) => {
networkConfig.shards = [parseInt(process.env.WAKU_SHARD, 10)];
}
// Inject network configuration as a global variable
const configScript = ` <script>window.__WAKU_NETWORK_CONFIG = ${JSON.stringify(networkConfig)};</script>`;
// Get lightpushnode configuration from environment
const lightpushNode = process.env.WAKU_LIGHTPUSH_NODE || null;
// Inject network configuration and lightpushnode as global variables
const configScript = ` <script>
window.__WAKU_NETWORK_CONFIG = ${JSON.stringify(networkConfig)};
window.__WAKU_LIGHTPUSH_NODE = ${JSON.stringify(lightpushNode)};
</script>`;
const originalPattern = ' <script type="module" src="./index.js"></script>';
const replacement = `${configScript}\n <script type="module" src="./index.js"></script>`;
@ -174,12 +180,13 @@ process.on("SIGTERM", (async () => {
}) as any);
/**
* Parse CLI arguments for cluster and shard configuration
* Parse CLI arguments for cluster, shard, and lightpushnode configuration
*/
function parseCliArgs() {
const args = process.argv.slice(2);
let clusterId: number | undefined;
let shard: number | undefined;
let lightpushNode: string | undefined;
for (const arg of args) {
if (arg.startsWith('--cluster-id=')) {
@ -194,10 +201,16 @@ function parseCliArgs() {
console.error('Invalid shard value. Must be a number.');
process.exit(1);
}
} else if (arg.startsWith('--lightpushnode=')) {
lightpushNode = arg.split('=')[1];
if (!lightpushNode || lightpushNode.trim() === '') {
console.error('Invalid lightpushnode value. Must be a valid multiaddr.');
process.exit(1);
}
}
}
return { clusterId, shard };
return { clusterId, shard, lightpushNode };
}
const isMainModule = process.argv[1] === fileURLToPath(import.meta.url);
@ -215,6 +228,10 @@ if (isMainModule) {
process.env.WAKU_SHARD = cliArgs.shard.toString();
console.log(`Using CLI shard: ${cliArgs.shard}`);
}
if (cliArgs.lightpushNode !== undefined) {
process.env.WAKU_LIGHTPUSH_NODE = cliArgs.lightpushNode;
console.log(`Using CLI lightpushnode: ${cliArgs.lightpushNode}`);
}
void startServer(port);
}

120
packages/browser-tests/web/index.ts
View File

@ -38,10 +38,16 @@ function makeSerializable(result: SDKProtocolResult): SerializableSDKProtocolRes
export class WakuHeadless {
waku: LightNode | null;
networkConfig: NetworkConfig;
constructor(networkConfig?: Partial<NetworkConfig>) {
lightpushNode: string | null;
constructor(networkConfig?: Partial<NetworkConfig>, lightpushNode?: string) {
this.waku = null as unknown as LightNode;
// Use provided config or defaults
this.networkConfig = this.buildNetworkConfig(networkConfig);
this.lightpushNode = lightpushNode || null;
if (this.lightpushNode) {
console.log(`Configured preferred lightpush node: ${this.lightpushNode}`);
}
}
/**
@ -204,11 +210,36 @@ export class WakuHeadless {
const encoder = this.waku.createEncoder({ contentTopic, pubsubTopic });
console.log("Encoder created with pubsubTopic:", encoder.pubsubTopic);
// Send the message using lightpush
const result = await lightPush.send(encoder, {
payload: processedPayload,
timestamp: new Date(),
});
// Send the message using lightpush with preferred peer if configured
let result;
if (this.lightpushNode) {
console.log(`Attempting to send via preferred lightpush node: ${this.lightpushNode}`);
try {
// Try to send to preferred peer first
const preferredPeerId = await this.getPeerIdFromMultiaddr(this.lightpushNode);
if (preferredPeerId) {
result = await lightPush.send(encoder, {
payload: processedPayload,
timestamp: new Date(),
}, { peerId: preferredPeerId });
console.log("✅ Message sent via preferred lightpush node");
} else {
throw new Error("Could not extract peer ID from preferred node address");
}
} catch (error) {
console.warn("Failed to send via preferred node, falling back to default:", error);
result = await lightPush.send(encoder, {
payload: processedPayload,
timestamp: new Date(),
});
}
} else {
result = await lightPush.send(encoder, {
payload: processedPayload,
timestamp: new Date(),
});
}
// Convert to serializable format for cross-context communication
const serializableResult = makeSerializable(result);
@ -332,9 +363,80 @@ export class WakuHeadless {
console.log("Starting Waku node...");
await this.waku.start();
console.log("Waku node started, peer ID:", this.waku.libp2p.peerId.toString());
// If a preferred lightpush node is configured, dial it
if (this.lightpushNode) {
await this.dialPreferredLightpushNode();
}
return { success: true };
}
/**
* Dial the preferred lightpush node if configured
*/
private async dialPreferredLightpushNode() {
if (!this.waku || !this.lightpushNode) {
return;
}
try {
console.log(`Dialing preferred lightpush node: ${this.lightpushNode}`);
await this.waku.dial(this.lightpushNode);
console.log(`Successfully connected to preferred lightpush node: ${this.lightpushNode}`);
} catch (error) {
console.warn(`Failed to dial preferred lightpush node ${this.lightpushNode}:`, error);
// Don't throw error - fallback to default peer discovery
}
}
/**
* Extract peer ID from multiaddr string
*/
private async getPeerIdFromMultiaddr(multiaddr: string): Promise<any | null> {
if (!this.waku) {
return null;
}
try {
// Check if this peer is already connected
const connectedPeers = this.waku.libp2p.getPeers();
// Try to match by the multiaddr - this is a simplified approach
// In a real implementation, you'd parse the multiaddr to extract the peer ID
for (const peerId of connectedPeers) {
try {
const peerInfo = await this.waku.libp2p.peerStore.get(peerId);
for (const addr of peerInfo.addresses) {
if (addr.multiaddr.toString().includes(multiaddr.split('/')[2])) {
console.log(`Found matching peer ID for ${multiaddr}: ${peerId.toString()}`);
return peerId;
}
}
} catch (e) {
// Continue searching
}
}
// If not found, try to extract from multiaddr format
// Format: /ip4/x.x.x.x/tcp/port/p2p/peerID
const parts = multiaddr.split('/');
const p2pIndex = parts.indexOf('p2p');
if (p2pIndex !== -1 && p2pIndex + 1 < parts.length) {
const peerIdString = parts[p2pIndex + 1];
console.log(`Extracted peer ID from multiaddr: ${peerIdString}`);
// For now, return as string - the actual implementation might need proper PeerId construction
return peerIdString;
}
console.warn(`Could not extract peer ID from multiaddr: ${multiaddr}`);
return null;
} catch (error) {
console.warn("Error extracting peer ID from multiaddr:", error);
return null;
}
}
async stopNode() {
if (!this.waku) {
throw new Error("Waku node not created");
@ -519,7 +621,11 @@ export class WakuHeadless {
// Check for global network configuration set by server
const globalNetworkConfig = (window as any).__WAKU_NETWORK_CONFIG;
const instance = new WakuHeadless(globalNetworkConfig);
// Check for global lightpushnode configuration set by server
const globalLightpushNode = (window as any).__WAKU_LIGHTPUSH_NODE;
const instance = new WakuHeadless(globalNetworkConfig, globalLightpushNode);
// @ts-ignore - will add proper typings in global.d.ts
(window as any).wakuApi = instance;