mirror of
https://github.com/logos-messaging/research.git
synced 2026-01-03 06:33:07 +00:00
135 lines
6.4 KiB
Markdown
135 lines
6.4 KiB
Markdown
# rln-delay-simulations
|
||
|
||
This folder contains two methods of simulations, that aim to estimate the latency of waku messages in the network:
|
||
* Method 1: Using `shadow`, which allows simulating hundreds of nodes in a single machine, considering network conditions but not CPU. See [report](https://github.com/waku-org/research/issues/42)
|
||
* Method 2: Using Digital Ocean, deploying real nodes in different locations in real machines with real network conditions, but due to cost limited to few nodes.
|
||
|
||
## Method 1: Shadow
|
||
|
||
This folder contains a `shadow` configuration to simulate `1000` `nwaku` nodes in an end to end setup:
|
||
* `nwaku` binaries are used, built with `make wakunode2` but with a minor modification, see [simulations](https://github.com/waku-org/nwaku/compare/master...simulations)
|
||
* `rln` is used with hardcoded static memberships, to avoid the sepolia node + contract, [see](https://raw.githubusercontent.com/waku-org/nwaku/master/waku/waku_rln_relay/constants.nim).
|
||
* Focused on measuring message propagation delays. Each message that is sent, encodes the timestamp when it was created.
|
||
* Requires significant resources to run (tested with 256 GB RAM)
|
||
* See simulation parameters: latency, bandwidth, amount of nodes, amount of publishers.
|
||
* Note that due to TCP flow control, when using big messages the first ones to arrive will show a higher delay. Filter them out to not bias the measurements.
|
||
|
||
### How to run
|
||
|
||
Get `nwaku` codebase and checkout to [simulations](https://github.com/waku-org/nwaku/tree/simulations) branch, build it and start the [shadow](https://github.com/shadow/shadow) simulation. Ensure `path` points to the `wakunode2` binary and you have enough resources.
|
||
|
||
```
|
||
git clone https://github.com/waku-org/nwaku.git
|
||
cd nwaku
|
||
git checkout simulations
|
||
make wakunode2
|
||
shadow shadow.yaml
|
||
```
|
||
|
||
### How to analyze
|
||
|
||
First check that the simulation finished ok. Check that the numbers match.
|
||
```
|
||
grep -nr 'ended_simulation' shadow.data | wc -l
|
||
# expected: 1000 (simulation finished ok in all nodes)
|
||
|
||
grep -nr 'tx_msg' shadow.data | wc -l
|
||
# expected: 10 (total of published messages)
|
||
|
||
grep -nr 'rx_msg' shadow.data | wc -l
|
||
# expected: 9990 (total rx messages)
|
||
```
|
||
|
||
Get metrics:
|
||
```
|
||
grep -nr 'rx_msg' shadow.data > latency.txt
|
||
grep -nr 'mesh_size' shadow.data > mesh_size.txt
|
||
```
|
||
|
||
Print results:
|
||
```
|
||
python analyze.py latency.txt "arrival_diff="
|
||
python analyze.py mesh_size.txt "mesh_size="
|
||
```
|
||
|
||
## Method 2: Digital Ocean
|
||
|
||
In this method we deploy real `nwaku` nodes at different locations with [some traces](https://github.com/waku-org/nwaku/compare/master...benchmark-latencies) that allow us to measure the propagation times of a given message across all nodes. For this experiment, 5
|
||
locations were selected:
|
||
* Frankfurt
|
||
* New York
|
||
* San Francisco
|
||
* Bangalore
|
||
* Singapore
|
||
|
||
Since deploying thousands of nodes would be costly, we connected the nodes in cascade:
|
||
`Singapore<->Bangalore<->San Francisco<->New York<->Frankfurt`
|
||
|
||
This forces a message to travel multiple hops. For example, a message introduced by the `Singapore` instance has to travel via `Bangalore` and `San Francisco` before reaching `New York`. This effectively simulates the existing hops in a real network.
|
||
The following commands allow to reproduce the setup. Its assumed that you have 5 different machines at 5 different locations and you can ssh into them:
|
||
|
||
In every machine, compile `wakunode2`
|
||
```
|
||
apt-get install build-essential git libpq5
|
||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||
source "$HOME/.cargo/env"
|
||
git clone https://github.com/waku-org/nwaku.git
|
||
cd nwaku
|
||
git checkout benchmark-latencies
|
||
make wakunode2
|
||
```
|
||
|
||
Start `Singapore` node. Note `rest` api is enabled. Set also the message size that you want. Set `--max-msg-size=600KB` if you want a bigger message size.
|
||
|
||
```
|
||
export MSG_SIZE_KB=100
|
||
./build/wakunode2 --rest --rest-address=0.0.0.0 --relay=true --topic=/waku/2/test --rln-relay=true --rln-relay-dynamic=false --rln-relay-membership-index=0 --nodekey=070a6101339f8e03a56bf21127dbbb0110b9b6efdb1e217115ed6d80da7a46d0
|
||
```
|
||
|
||
Connect `Bangalore`<->`Singapore`
|
||
```
|
||
./build/wakunode2 --relay=true --topic=/waku/2/test --rln-relay=true --rln-relay-dynamic=false --rln-relay-membership-index=1 --nodekey=e9c166557cf6cf1d0fc6a4b1bb98e417a6de6c361b228dea72d54ffe4442a115 --staticnode=/ip4/SINGAPORE_IP/tcp/60000/p2p/16Uiu2HAmU3GnnKHPLJFWDLGMEt1mNDAFmaKWUdkR9gWutaLbk2xx
|
||
```
|
||
|
||
|
||
Connect `San Francisco`<->`Bangalore`
|
||
```
|
||
./build/wakunode2 --relay=true --topic=/waku/2/test --rln-relay=true --rln-relay-dynamic=false --rln-relay-membership-index=2 --nodekey=f9a4f0f889b6dbf55d6b32bb8a85c418df01f013cebcd23efd8a250df65d9337 --staticnode=/ip4/BANGALORE_IP/tcp/60000/p2p/16Uiu2HAmSDAp4VrbKQPStDLg7rc38JJR3zE5mJcFieAGJLBrCFCy
|
||
|
||
```
|
||
|
||
Connect `New York`<->`San Francisco`
|
||
```
|
||
./build/wakunode2 --relay=true --topic=/waku/2/test --rln-relay=true --rln-relay-dynamic=false --rln-relay-membership-index=3 --nodekey=100a04176710aabdec3258e1b6cdfbbdf602af36ea2311415ae7504bddd86cac --staticnode=/ip4/SANFRANCISCO_IP/tcp/60000/p2p/16Uiu2HAm8zWqrWRp6typPSdL7nqBRGbabH87vmkzN6A3McaGDj3C
|
||
|
||
```
|
||
|
||
Connect `Frankfurt`<->`NewYork`
|
||
```
|
||
./build/wakunode2 --relay=true --topic=/waku/2/test --rln-relay=true --rln-relay-dynamic=false --rln-relay-membership-index=4 --nodekey=eb131c2ee17807042f5051b6d7b9fbbbdc83369f28315157d8401fa13bf2b88f --staticnode=/ip4/NEW_YORK_IP/tcp/60000/p2p/16Uiu2HAmJdukvEFU1LhCQHGNcFviWMJh95PU4vMoun2uUvWtaWQL
|
||
```
|
||
|
||
|
||
Now you can inject a message via the `rest` API of the node in `Singapore`. This message will travel all the way to `Frankfurt` in 4 hops.
|
||
```
|
||
curl -X POST "http://SINGAPORE_IP:8645/relay/v1/messages/%2Fwaku%2F2%2Ftest" \
|
||
-H "content-type: application/json" \
|
||
-d '{"payload":"dontcare","contentTopic":"string"}'
|
||
```
|
||
|
||
If you check the logs of every machine, you will find the timestamp of when each node received the message.
|
||
|
||
Measuring the latency between each node can be interesting. It can be done with the [wakucanary](https://github.com/waku-org/nwaku/tree/master/apps/wakucanary) tool as follows.
|
||
|
||
Run `nwaku` in one node:
|
||
```
|
||
docker run -p 60000:60000 harbor.status.im/wakuorg/nwaku:v0.25.0 --relay=true --nodekey=b03f13c0e075643e39b19ddef9206f868e94b759c7e847383b455f8d1991e695
|
||
```
|
||
|
||
And ping from other node. `REPLACE_IP` by the nodes public ip. The ping in `ms` will be displayed in the logs, measured using the `ping` protocol from `libp2p`.
|
||
```
|
||
docker run wakucanary:a1d5cbd -a=/ip4/REPLACE_IP/tcp/60000/p2p/16Uiu2HAmBxK6wbHqGtTKgEuYtuCxQCxGRKcu7iZeouEpaUr7ewwK -p=relay --ping
|
||
```
|
||
|
||
|