logos-co
/
Waku-topology-test
mirror of https://github.com/logos-co/Waku-topology-test.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

typer, json/toml, more network/node type, refactor

This commit is contained in:
0xFugue 2022-12-29 21:34:01 +05:30
parent d677a4c78d
commit 6e96757d10
3 changed files with 186 additions and 151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

49
Readme.md
View File

@ -1,45 +1,12 @@
This repo contains the scripts to generate different network models for wakukurtosis runs.
This repo contains scripts to generate network models (in JSON) and waku configuration files (in TOMLs) for wakukurtosis runs.
## run_kurtosis_tests.sh
run_kurtosis_tests.sh will run kurtosis on a set of json files in a directory. It requires two arguments. First is a directory containing json files; other file types in the directory are ignored. Second is the github root/prefix of the kurtosis module you run the tests under.</br>
> usage: ./run_kurtosis_tests.sh <input_dir> <repo_prefix> </br>
Running this script is somewhat complicated; so follow the following instructions to a dot. You **WILL** require the main.star provided here. The main.star reads a input json and instantiates Waku nodes accordingly. The runs are repeated for each of the input json files under the specified directory.
#### step 0)
symlink run_kurtosis_tests.sh to the root directory of your kurtosis module.</br>
#### step 1)
backup the your kurtosis module's own main.star. copy the main.star provided here to the root directory of your kurtosis module.</br>
!!! WARNING: symlinking the main.star will NOT work !!!</br>
#### step 3)
put all the json files you want to use in a directory. Call it *Foo*</br>
#### step 3)
copy the *Foo* directory to the root of your kurtosis module</br>
!!! WARNING: symlinking the directory will NOT work !!!</br>
#### step 4)
run this script in the root directory of the kurtosis module. provide the directory (*Foo*) and the github root/prefix of the kurtosis module as arguments to the script</br>
## gen_jsons.sh
gen_jsons.sh can generate given number of Waku networs and outputs them to a directory. Please make sure that the output directory exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
> usage: ./gen_jsons.sh <output_dir> <#json files needed> </br>
## generate_network.py
generate_network.py can generate networks with specified number of nodes and topics. the network types currently supported is "configuration_model" and more are on the way. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
generate_network.py generates one network and per-node configuration files. The tool is configurable with specified number of nodes, topics, network types, node types. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
> usage: generate_network [-h] [-o <file_name>] [-n <#nodes>] [-t <#topics>]
[-T <type>] <br>
>> </br>
>> Generates and outputs the Waku network conforming to input parameters<//br>
>> </br>
>> optional arguments:</br>
>> &emsp; -h, --help show this help message and exit</br>
>> &emsp; -o <file_name>, --output <file_name> output json filename for the Waku network </br>
>> &emsp; -n <#nodes>, --numnodes <#nodes> number of nodes in the Waku network </br>
>> &emsp; -t <#topics>, --numtopics <#topics> number of topics in the Waku network </br>
>> &emsp; -T <type>, --type <type> network type for the Waku network </br>
>> &emsp; -p <#partitions>, --numparts <#partitions> number of partitions in the Waku network</br>
>></br>
>>The defaults are: -o "Topology.json"; -n 1; -t 1; -p 1; -T "configuration_model"</br>
> usage: $./generate_network --help
## batch_gen.sh
batch_gen.sh can generate given number of Waku networks and outputs them to a directory. Please make sure that the output directory does not exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
> usage: $./batch_gen.sh <output-dir> <#number of networks needed> </br>

47
batch_gen.sh Executable file
View File

@ -0,0 +1,47 @@
#!/bin/sh
#MAX and MIN for topics and num nodes
MIN=5
MAX=100
#requires bc
getrand(){
orig=$(od -An -N1 -i /dev/urandom)
range=`echo "$MIN + ($orig % ($MAX - $MIN + 1))" | bc`
RANDOM=$range
}
getrand1(){
orig=$(od -An -N1 -i /dev/urandom)
range=`echo "$MIN + ($orig % ($MAX - $MIN + 1))" | bc`
return range
#getrand1 # call the fun and use the return value
#n=$?
}
if [ "$#" -ne 2 ] || [ $2 -le 0 ] ; then
echo "usage: $0 <output dir> <#json files needed>" >&2
exit 1
fi
path=$1
nfiles=$2
mkdir -p $path
echo "Ok, will generate $nfiles networks & put them under '$path'."
nwtype="NEWMANWATTSSTROGATZ"
nodetype="DESKTOP"
for i in $(seq $nfiles)
do
getrand
n=$((RANDOM+1))
getrand
t=$((RANDOM+1))
dirname="$path/$i/Waku"
mkdir "$path/$i"
echo "Generating ./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1"
$(./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1)
done

239
generate_network.py
View File

@ -4,79 +4,68 @@ import matplotlib.pyplot as plt
import networkx as nx
import random, math
import json
import argparse, os, sys
import sys, os
import string
import typer
from enum import Enum
# Consts
class nw_types(Enum):
configmodel = "CONFIGMODEL"
scalefree = "SCALEFREE" # power law
newmanwattsstrogatz = "NEWMANWATTSSTROGATZ" # mesh, smallworld
barbell = "BARBELL" # partition
balancedtree = "BALANCEDTREE" # committees?
star = "STAR" # spof
class node_types(Enum):
desktop = "DESKTOP"
mobile = "MOBILE"
nw_fname = "network_data.json"
prefix = "waku_"
### I/O related fns ###########################################################
# Dump to a json file
def write_json(filename, data_2_dump):
json.dump(data_2_dump, open(filename,'w'), indent=2)
def write_json(dirname, json_dump):
fname = os.path.join(dirname, nw_fname)
json.dump(json_dump, open(fname,'w'), indent=2)
# has trouble with non-integer/non-hashable keys
def read_json(filename):
with open(filename) as f:
jdata = json.load(f)
return nx.node_link_graph(jdata)
def write_toml(dirname, node_name, toml):
fname = os.path.join(dirname, node_name+ ".toml")
f = open(fname, 'w')
f.write(toml)
f.close()
# Draw the network and output the image to a file
def draw(fname, H):
def draw(dirname, H):
nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
fname = os.path.join(dirname, nw_fname)
plt.savefig(os.path.splitext(fname)[0] + ".png", format="png")
plt.show()
# Initialize parser, set the defaults, and extract the options
def get_options():
parser = argparse.ArgumentParser(
prog = 'generate_network',
description = '''Generates and outputs
the Waku network conforming to input parameters''',
epilog = '''Defaults: -o "Topology.json";
-n 1; -t 1; -p 1; -T "configuration_model"
Supported nw types "configuration_model", "scalefree",
"newman_watts_strogatz"''')
parser.add_argument("-o", "--output",
default='Topology.json', dest='fname',
help='output json filename for the Waku network',
type=str, metavar='<file_name>')
parser.add_argument("-n", "--numnodes",
default=1, dest='num_nodes',
help='number of nodes in the Waku network',
type=int, metavar='<#nodes>')
parser.add_argument("-t", "--numtopics",
default=1, dest='num_topics',
help='number of topics in the Waku network',
type=int, metavar='<#topics>')
parser.add_argument("-T", "--type",
default="configuration_model", dest='nw_type',
help='network type of the Waku network',
type=str, metavar='<type>')
parser.add_argument("-p", "--numparts",
default=1, dest='num_partitions',
help='The number of partitions in the Waku network',
type=int, metavar='<#partitions>')
# parser.add_argument("-e", "--numedges",
# default=1, dest='num_edges',
# help='The number of edges in the Waku network',
# type=int, metavar='#edges>')
return parser.parse_args()
# Has trouble with non-integer/non-hashable keys
def read_json(fname):
with open(fname) as f:
jdata = json.load(f)
return nx.node_link_graph(jdata)
# Generate a random string (UC chars) of len n
def generate_topic_string(n):
rs = ""
for _ in range(n):
r = random.randint(65, 65 + 26 - 1) # generate a random UC char
rs += (chr(r)) # append the char generated
return rs
### topics related fns ###########################################################
# Generate a random string of upper case chars
def generate_random_string(n):
return "".join(random.choice(string.ascii_uppercase) for _ in range(n))
# Generate the topics - UC chars prefixed by "topic"
# Generate the topics - topic followed by random UC chars - Eg, topic_XY"
def generate_topics(num_topics):
topics = []
base = 26
topic_len = int(math.log(num_topics)/math.log(base)) + 1
topics = {i: f"topic_{generate_topic_string(topic_len)}" for i in range(num_topics)}
topic_len = int(math.log(num_topics)/math.log(26)) + 1 # base is 26 - upper case letters
topics = {i: f"topic_{generate_random_string(topic_len)}" for i in range(num_topics)}
return topics
@ -91,12 +80,13 @@ def get_random_sublist(topics):
return sublist
### network processing related fns ###########################################################
# Network Types
# https://networkx.org/documentation/stable/reference/generated/networkx.generators.degree_seq.configuration_model.html
def generate_config_model(n):
#degrees = nx.random_powerlaw_tree_sequence(n, tries=10000)
degrees = [random.randint(1, n) for i in range(n)]
if (sum(degrees)) % 2 != 0: # adjust the degree to even
if (sum(degrees)) % 2 != 0: # adjust the degree to be even
degrees[-1] += 1
return nx.configuration_model(degrees) # generate the graph
@ -105,79 +95,110 @@ def generate_scalefree_graph(n):
return nx.scale_free_graph(n)
# n must be larger than k
# n must be larger than k=D=3
def generate_newman_watts_strogatz_graph(n):
return nx.newman_watts_strogatz_graph(n, 12, 0.5)
return nx.newman_watts_strogatz_graph(n, 3, 0.5)
def generate_barbell_graph(n):
return nx.barbell_graph(int(n/2), 1)
def generate_balanced_tree(n, fanout=3):
height = int(math.log(n)/math.log(fanout))
return nx.balanced_tree(fanout, height)
def generate_star_graph(n):
return nx.star_graph(n)
# Generate the network from nw type
def generate_network(num_nodes, nw_type, prefix):
def generate_network(num_nodes, nw_type):
G = nx.empty_graph()
if nw_type == "configuration_model":
if nw_type == nw_types.configmodel:
G = generate_config_model(num_nodes)
elif nw_type == "scalefree":
elif nw_type == nw_types.scalefree:
G = generate_scalefree_graph(num_nodes)
elif nw_type == "newman_watts_strogatz":
elif nw_type == nw_types.newmanwattsstrogatz:
G = generate_newman_watts_strogatz_graph(num_nodes)
elif nw_type == nw_types.barbell:
G = generate_barbell_graph(num_nodes)
elif nw_type == nw_types.balancedtree:
G = generate_balanced_tree(num_nodes)
elif nw_type == nw_types.star:
G = generate_star_graph(num_nodes)
else:
print(nw_type +": Unsupported network type")
sys.exit(1)
H = postprocess_network(G, prefix)
return H
return postprocess_network(G)
# used by generate_dump_data, *ought* to be global to handle partitions
ports_shifted = 0
def postprocess_network(G, prefix):
# Label the generated network with prefix
def postprocess_network(G):
G = nx.Graph(G) # prune out parallel/multi edges
G.remove_edges_from(nx.selfloop_edges(G)) # Removing self-loops
# Labeling nodes to match waku containers
G.remove_edges_from(nx.selfloop_edges(G)) # remove the self-loops
mapping = {i: f"{prefix}{i}" for i in range(len(G))}
return nx.relabel_nodes(G, mapping)
return nx.relabel_nodes(G, mapping) # label the nodes
# Generate dump data from the network and topics
def generate_dump_data(H, topics):
data_to_dump = {}
global ports_shifted
for node in H.nodes:
data_to_dump[node] = {}
data_to_dump[node]["ports-shift"] = ports_shifted
ports_shifted += 1
data_to_dump[node]["topics"] = get_random_sublist(topics)
data_to_dump[node]["static-nodes"] = []
for edge in H.edges(node):
data_to_dump[node]["static-nodes"].append(edge[1])
return data_to_dump
### file format related fns ###########################################################
#Generate per node toml configs
def generate_toml(topics, node_type=node_types.desktop):
topic_str = " ". join(get_random_sublist(topics)) # topics as a space separated string
if node_type == node_type.desktop:
toml = "rpc-admin = true\nkeep-alive = true\n"
elif node_type == node_type.mobile:
toml = "rpc-admin = true\nkeep-alive = true\n"
else:
print(node_type +": Unsupported node type")
sys.exit(1)
toml += f"topics = \"{topic_str}\"\n"
return toml
def main():
#extract the CLI arguments and assign params
options = get_options()
fname = options.fname
num_nodes = options.num_nodes
num_topics = options.num_topics
nw_type = options.nw_type
prefix = "waku_"
num_partitions = options.num_partitions
#num_edges = options.num_edges ## need to control num_edges?
if num_partitions > 1:
print("-p",num_partitions,
"Sorry, we do not yet support partitions")
# Generates network-wide json and per-node toml and writes them
def generate_and_write_files(dirname, num_topics, H):
if not os.path.exists(dirname):
os.mkdir(dirname)
elif not os.path.isfile(dirname) and os.listdir(dirname):
print(dirname +": exists and is not empty")
sys.exit(1)
elif os.path.isfile(dirname):
print(dirname +": exists and is not a directory")
sys.exit(1)
# Generate the network and postprocess it
H = generate_network(num_nodes, nw_type, prefix)
# Generate the topics
topics = generate_topics(num_topics)
# Generate the dump data
dump_data = generate_dump_data(H, topics)
# Dump the network in a json file
write_json(fname, dump_data)
# Display the graph
draw(fname, H)
json_dump = {}
for node in H.nodes:
write_toml(dirname, node, generate_toml(topics)) # per node toml
json_dump[node] = {}
json_dump[node]["static-nodes"] = []
for edge in H.edges(node):
json_dump[node]["static-nodes"].append(edge[1])
write_json(dirname, json_dump) # network wide json
### the main ###########################################################
def main(
dirname: str = "Waku", num_nodes: int = 3, num_topics: int = 1,
nw_type: nw_types = "NEWMANWATTSSTROGATZ",
node_type: node_types = "DESKTOP",
num_partitions: int = 1):
if num_partitions > 1:
print("-p",num_partitions, "Sorry, we do not yet support partitions")
sys.exit(1)
# Generate the network and do post-process
G = generate_network(num_nodes, nw_type)
postprocess_network(G)
# Generate file format specific data structs and write the files; optionally, draw the network
generate_and_write_files(dirname, num_topics, G)
draw(dirname, G)
if __name__ == "__main__":
main()
typer.run(main)