logos-co
/
Waku-topology-test
mirror of https://github.com/logos-co/Waku-topology-test.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

typer, json/toml, more network/node type, refactor

This commit is contained in:
0xFugue 2022-12-29 21:34:01 +05:30
parent d677a4c78d
commit 6e96757d10
3 changed files with 186 additions and 151 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

49
Readme.md
View File

@ -1,45 +1,12 @@
This repo contains the scripts to generate different network models for wakukurtosis runs. This repo contains scripts to generate network models (in JSON) and waku configuration files (in TOMLs) for wakukurtosis runs.
## run_kurtosis_tests.sh
run_kurtosis_tests.sh will run kurtosis on a set of json files in a directory. It requires two arguments. First is a directory containing json files; other file types in the directory are ignored. Second is the github root/prefix of the kurtosis module you run the tests under.</br>
> usage: ./run_kurtosis_tests.sh <input_dir> <repo_prefix> </br>
Running this script is somewhat complicated; so follow the following instructions to a dot. You **WILL** require the main.star provided here. The main.star reads a input json and instantiates Waku nodes accordingly. The runs are repeated for each of the input json files under the specified directory.
#### step 0)
symlink run_kurtosis_tests.sh to the root directory of your kurtosis module.</br>
#### step 1)
backup the your kurtosis module's own main.star. copy the main.star provided here to the root directory of your kurtosis module.</br>
!!! WARNING: symlinking the main.star will NOT work !!!</br>
#### step 3)
put all the json files you want to use in a directory. Call it *Foo*</br>
#### step 3)
copy the *Foo* directory to the root of your kurtosis module</br>
!!! WARNING: symlinking the directory will NOT work !!!</br>
#### step 4)
run this script in the root directory of the kurtosis module. provide the directory (*Foo*) and the github root/prefix of the kurtosis module as arguments to the script</br>
## gen_jsons.sh
gen_jsons.sh can generate given number of Waku networs and outputs them to a directory. Please make sure that the output directory exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
> usage: ./gen_jsons.sh <output_dir> <#json files needed> </br>
## generate_network.py ## generate_network.py
generate_network.py can generate networks with specified number of nodes and topics. the network types currently supported is "configuration_model" and more are on the way. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph. generate_network.py generates one network and per-node configuration files. The tool is configurable with specified number of nodes, topics, network types, node types. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
> usage: generate_network [-h] [-o <file_name>] [-n <#nodes>] [-t <#topics>] > usage: $./generate_network --help
[-T <type>] <br>
>> </br> ## batch_gen.sh
>> Generates and outputs the Waku network conforming to input parameters<//br> batch_gen.sh can generate given number of Waku networks and outputs them to a directory. Please make sure that the output directory does not exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
>> </br>
>> optional arguments:</br> > usage: $./batch_gen.sh <output-dir> <#number of networks needed> </br>
>> &emsp; -h, --help show this help message and exit</br>
>> &emsp; -o <file_name>, --output <file_name> output json filename for the Waku network </br>
>> &emsp; -n <#nodes>, --numnodes <#nodes> number of nodes in the Waku network </br>
>> &emsp; -t <#topics>, --numtopics <#topics> number of topics in the Waku network </br>
>> &emsp; -T <type>, --type <type> network type for the Waku network </br>
>> &emsp; -p <#partitions>, --numparts <#partitions> number of partitions in the Waku network</br>
>></br>
>>The defaults are: -o "Topology.json"; -n 1; -t 1; -p 1; -T "configuration_model"</br>

47
batch_gen.sh Executable file
View File

@ -0,0 +1,47 @@
#!/bin/sh
#MAX and MIN for topics and num nodes
MIN=5
MAX=100
#requires bc
getrand(){
orig=$(od -An -N1 -i /dev/urandom)
range=`echo "$MIN + ($orig % ($MAX - $MIN + 1))" | bc`
RANDOM=$range
}
getrand1(){
orig=$(od -An -N1 -i /dev/urandom)
range=`echo "$MIN + ($orig % ($MAX - $MIN + 1))" | bc`
return range
#getrand1 # call the fun and use the return value
#n=$?
}
if [ "$#" -ne 2 ] || [ $2 -le 0 ] ; then
echo "usage: $0 <output dir> <#json files needed>" >&2
exit 1
fi
path=$1
nfiles=$2
mkdir -p $path
echo "Ok, will generate $nfiles networks & put them under '$path'."
nwtype="NEWMANWATTSSTROGATZ"
nodetype="DESKTOP"
for i in $(seq $nfiles)
do
getrand
n=$((RANDOM+1))
getrand
t=$((RANDOM+1))
dirname="$path/$i/Waku"
mkdir "$path/$i"
echo "Generating ./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1"
$(./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1)
done

241
generate_network.py
View File

@ -4,79 +4,68 @@ import matplotlib.pyplot as plt
import networkx as nx import networkx as nx
import random, math import random, math
import json import json
import argparse, os, sys import sys, os
import string
import typer
from enum import Enum
# Consts
class nw_types(Enum):
configmodel = "CONFIGMODEL"
scalefree = "SCALEFREE" # power law
newmanwattsstrogatz = "NEWMANWATTSSTROGATZ" # mesh, smallworld
barbell = "BARBELL" # partition
balancedtree = "BALANCEDTREE" # committees?
star = "STAR" # spof
class node_types(Enum):
desktop = "DESKTOP"
mobile = "MOBILE"
nw_fname = "network_data.json"
prefix = "waku_"
### I/O related fns ###########################################################
# Dump to a json file # Dump to a json file
def write_json(filename, data_2_dump): def write_json(dirname, json_dump):
json.dump(data_2_dump, open(filename,'w'), indent=2) fname = os.path.join(dirname, nw_fname)
json.dump(json_dump, open(fname,'w'), indent=2)
# has trouble with non-integer/non-hashable keys def write_toml(dirname, node_name, toml):
def read_json(filename): fname = os.path.join(dirname, node_name+ ".toml")
with open(filename) as f: f = open(fname, 'w')
jdata = json.load(f) f.write(toml)
return nx.node_link_graph(jdata) f.close()
# Draw the network and output the image to a file # Draw the network and output the image to a file
def draw(fname, H): def draw(dirname, H):
nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True) nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
fname = os.path.join(dirname, nw_fname)
plt.savefig(os.path.splitext(fname)[0] + ".png", format="png") plt.savefig(os.path.splitext(fname)[0] + ".png", format="png")
plt.show() plt.show()
# Initialize parser, set the defaults, and extract the options # Has trouble with non-integer/non-hashable keys
def get_options(): def read_json(fname):
parser = argparse.ArgumentParser( with open(fname) as f:
prog = 'generate_network', jdata = json.load(f)
description = '''Generates and outputs return nx.node_link_graph(jdata)
the Waku network conforming to input parameters''',
epilog = '''Defaults: -o "Topology.json";
-n 1; -t 1; -p 1; -T "configuration_model"
Supported nw types "configuration_model", "scalefree",
"newman_watts_strogatz"''')
parser.add_argument("-o", "--output",
default='Topology.json', dest='fname',
help='output json filename for the Waku network',
type=str, metavar='<file_name>')
parser.add_argument("-n", "--numnodes",
default=1, dest='num_nodes',
help='number of nodes in the Waku network',
type=int, metavar='<#nodes>')
parser.add_argument("-t", "--numtopics",
default=1, dest='num_topics',
help='number of topics in the Waku network',
type=int, metavar='<#topics>')
parser.add_argument("-T", "--type",
default="configuration_model", dest='nw_type',
help='network type of the Waku network',
type=str, metavar='<type>')
parser.add_argument("-p", "--numparts",
default=1, dest='num_partitions',
help='The number of partitions in the Waku network',
type=int, metavar='<#partitions>')
# parser.add_argument("-e", "--numedges",
# default=1, dest='num_edges',
# help='The number of edges in the Waku network',
# type=int, metavar='#edges>')
return parser.parse_args()
# Generate a random string (UC chars) of len n ### topics related fns ###########################################################
def generate_topic_string(n):
rs = "" # Generate a random string of upper case chars
for _ in range(n): def generate_random_string(n):
r = random.randint(65, 65 + 26 - 1) # generate a random UC char return "".join(random.choice(string.ascii_uppercase) for _ in range(n))
rs += (chr(r)) # append the char generated
return rs
# Generate the topics - UC chars prefixed by "topic" # Generate the topics - topic followed by random UC chars - Eg, topic_XY"
def generate_topics(num_topics): def generate_topics(num_topics):
topics = [] topic_len = int(math.log(num_topics)/math.log(26)) + 1 # base is 26 - upper case letters
base = 26 topics = {i: f"topic_{generate_random_string(topic_len)}" for i in range(num_topics)}
topic_len = int(math.log(num_topics)/math.log(base)) + 1
topics = {i: f"topic_{generate_topic_string(topic_len)}" for i in range(num_topics)}
return topics return topics
@ -91,12 +80,13 @@ def get_random_sublist(topics):
return sublist return sublist
### network processing related fns ###########################################################
# Network Types # Network Types
# https://networkx.org/documentation/stable/reference/generated/networkx.generators.degree_seq.configuration_model.html
def generate_config_model(n): def generate_config_model(n):
#degrees = nx.random_powerlaw_tree_sequence(n, tries=10000) #degrees = nx.random_powerlaw_tree_sequence(n, tries=10000)
degrees = [random.randint(1, n) for i in range(n)] degrees = [random.randint(1, n) for i in range(n)]
if (sum(degrees)) % 2 != 0: # adjust the degree to even if (sum(degrees)) % 2 != 0: # adjust the degree to be even
degrees[-1] += 1 degrees[-1] += 1
return nx.configuration_model(degrees) # generate the graph return nx.configuration_model(degrees) # generate the graph
@ -105,79 +95,110 @@ def generate_scalefree_graph(n):
return nx.scale_free_graph(n) return nx.scale_free_graph(n)
# n must be larger than k # n must be larger than k=D=3
def generate_newman_watts_strogatz_graph(n): def generate_newman_watts_strogatz_graph(n):
return nx.newman_watts_strogatz_graph(n, 12, 0.5) return nx.newman_watts_strogatz_graph(n, 3, 0.5)
def generate_barbell_graph(n):
return nx.barbell_graph(int(n/2), 1)
def generate_balanced_tree(n, fanout=3):
height = int(math.log(n)/math.log(fanout))
return nx.balanced_tree(fanout, height)
def generate_star_graph(n):
return nx.star_graph(n)
# Generate the network from nw type # Generate the network from nw type
def generate_network(num_nodes, nw_type, prefix): def generate_network(num_nodes, nw_type):
G = nx.empty_graph() G = nx.empty_graph()
if nw_type == "configuration_model": if nw_type == nw_types.configmodel:
G = generate_config_model(num_nodes) G = generate_config_model(num_nodes)
elif nw_type == "scalefree": elif nw_type == nw_types.scalefree:
G = generate_scalefree_graph(num_nodes) G = generate_scalefree_graph(num_nodes)
elif nw_type == "newman_watts_strogatz": elif nw_type == nw_types.newmanwattsstrogatz:
G = generate_newman_watts_strogatz_graph(num_nodes) G = generate_newman_watts_strogatz_graph(num_nodes)
elif nw_type == nw_types.barbell:
G = generate_barbell_graph(num_nodes)
elif nw_type == nw_types.balancedtree:
G = generate_balanced_tree(num_nodes)
elif nw_type == nw_types.star:
G = generate_star_graph(num_nodes)
else: else:
print(nw_type +": Unsupported network type") print(nw_type +": Unsupported network type")
sys.exit(1) sys.exit(1)
H = postprocess_network(G, prefix) return postprocess_network(G)
return H
# used by generate_dump_data, *ought* to be global to handle partitions # Label the generated network with prefix
ports_shifted = 0 def postprocess_network(G):
def postprocess_network(G, prefix): G = nx.Graph(G) # prune out parallel/multi edges
G = nx.Graph(G) # prune out parallel/multi edges G.remove_edges_from(nx.selfloop_edges(G)) # remove the self-loops
G.remove_edges_from(nx.selfloop_edges(G)) # Removing self-loops
# Labeling nodes to match waku containers
mapping = {i: f"{prefix}{i}" for i in range(len(G))} mapping = {i: f"{prefix}{i}" for i in range(len(G))}
return nx.relabel_nodes(G, mapping) return nx.relabel_nodes(G, mapping) # label the nodes
# Generate dump data from the network and topics ### file format related fns ###########################################################
def generate_dump_data(H, topics):
data_to_dump = {} #Generate per node toml configs
global ports_shifted def generate_toml(topics, node_type=node_types.desktop):
for node in H.nodes: topic_str = " ". join(get_random_sublist(topics)) # topics as a space separated string
data_to_dump[node] = {} if node_type == node_type.desktop:
data_to_dump[node]["ports-shift"] = ports_shifted toml = "rpc-admin = true\nkeep-alive = true\n"
ports_shifted += 1 elif node_type == node_type.mobile:
data_to_dump[node]["topics"] = get_random_sublist(topics) toml = "rpc-admin = true\nkeep-alive = true\n"
data_to_dump[node]["static-nodes"] = [] else:
for edge in H.edges(node): print(node_type +": Unsupported node type")
data_to_dump[node]["static-nodes"].append(edge[1]) sys.exit(1)
return data_to_dump toml += f"topics = \"{topic_str}\"\n"
return toml
def main(): # Generates network-wide json and per-node toml and writes them
#extract the CLI arguments and assign params def generate_and_write_files(dirname, num_topics, H):
options = get_options() if not os.path.exists(dirname):
fname = options.fname os.mkdir(dirname)
num_nodes = options.num_nodes elif not os.path.isfile(dirname) and os.listdir(dirname):
num_topics = options.num_topics print(dirname +": exists and is not empty")
nw_type = options.nw_type sys.exit(1)
prefix = "waku_" elif os.path.isfile(dirname):
num_partitions = options.num_partitions print(dirname +": exists and is not a directory")
#num_edges = options.num_edges ## need to control num_edges?
if num_partitions > 1:
print("-p",num_partitions,
"Sorry, we do not yet support partitions")
sys.exit(1) sys.exit(1)
# Generate the network and postprocess it
H = generate_network(num_nodes, nw_type, prefix)
# Generate the topics
topics = generate_topics(num_topics) topics = generate_topics(num_topics)
# Generate the dump data json_dump = {}
dump_data = generate_dump_data(H, topics) for node in H.nodes:
# Dump the network in a json file write_toml(dirname, node, generate_toml(topics)) # per node toml
write_json(fname, dump_data) json_dump[node] = {}
# Display the graph json_dump[node]["static-nodes"] = []
draw(fname, H) for edge in H.edges(node):
json_dump[node]["static-nodes"].append(edge[1])
write_json(dirname, json_dump) # network wide json
### the main ###########################################################
def main(
dirname: str = "Waku", num_nodes: int = 3, num_topics: int = 1,
nw_type: nw_types = "NEWMANWATTSSTROGATZ",
node_type: node_types = "DESKTOP",
num_partitions: int = 1):
if num_partitions > 1:
print("-p",num_partitions, "Sorry, we do not yet support partitions")
sys.exit(1)
# Generate the network and do post-process
G = generate_network(num_nodes, nw_type)
postprocess_network(G)
# Generate file format specific data structs and write the files; optionally, draw the network
generate_and_write_files(dirname, num_topics, G)
draw(dirname, G)
if __name__ == "__main__": if __name__ == "__main__":
main() typer.run(main)