Added typer, combined JSON/TOMLs generations; added network/node types; refactored (#4)

* typer, json/toml, more network/node type, refactor

* cleanup

* PR#4 mods

* typer + enums + switch

* gitignore & minor edits
This commit is contained in:
0xFugue 2022-12-30 19:17:21 +05:30 committed by GitHub
parent d677a4c78d
commit 87433027be
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 168 additions and 244 deletions

4
.gitignore vendored
View File

@ -3,6 +3,10 @@
*.swo
*~
# dirs
Waku
WakuNetwork
# local json
Topology.*
topology.*

View File

@ -1,45 +1,12 @@
This repo contains the scripts to generate different network models for wakukurtosis runs.
This repo contains scripts to generate network models (in JSON) and waku configuration files (in TOMLs) for wakukurtosis runs.
## run_kurtosis_tests.sh
run_kurtosis_tests.sh will run kurtosis on a set of json files in a directory. It requires two arguments. First is a directory containing json files; other file types in the directory are ignored. Second is the github root/prefix of the kurtosis module you run the tests under.</br>
> usage: ./run_kurtosis_tests.sh <input_dir> <repo_prefix> </br>
Running this script is somewhat complicated; so follow the following instructions to a dot. You **WILL** require the main.star provided here. The main.star reads a input json and instantiates Waku nodes accordingly. The runs are repeated for each of the input json files under the specified directory.
#### step 0)
symlink run_kurtosis_tests.sh to the root directory of your kurtosis module.</br>
#### step 1)
backup the your kurtosis module's own main.star. copy the main.star provided here to the root directory of your kurtosis module.</br>
!!! WARNING: symlinking the main.star will NOT work !!!</br>
#### step 3)
put all the json files you want to use in a directory. Call it *Foo*</br>
#### step 3)
copy the *Foo* directory to the root of your kurtosis module</br>
!!! WARNING: symlinking the directory will NOT work !!!</br>
#### step 4)
run this script in the root directory of the kurtosis module. provide the directory (*Foo*) and the github root/prefix of the kurtosis module as arguments to the script</br>
## gen_jsons.sh
gen_jsons.sh can generate given number of Waku networs and outputs them to a directory. Please make sure that the output directory exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
> usage: ./gen_jsons.sh <output_dir> <#json files needed> </br>
## generate_network.py
generate_network.py can generate networks with specified number of nodes and topics. the network types currently supported is "configuration_model" and more are on the way. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
generate_network.py generates one network and per-node configuration files. The tool is configurable with specified number of nodes, topics, network types, node types. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
> usage: generate_network [-h] [-o <file_name>] [-n <#nodes>] [-t <#topics>]
[-T <type>] <br>
>> </br>
>> Generates and outputs the Waku network conforming to input parameters<//br>
>> </br>
>> optional arguments:</br>
>> &emsp; -h, --help show this help message and exit</br>
>> &emsp; -o <file_name>, --output <file_name> output json filename for the Waku network </br>
>> &emsp; -n <#nodes>, --numnodes <#nodes> number of nodes in the Waku network </br>
>> &emsp; -t <#topics>, --numtopics <#topics> number of topics in the Waku network </br>
>> &emsp; -T <type>, --type <type> network type for the Waku network </br>
>> &emsp; -p <#partitions>, --numparts <#partitions> number of partitions in the Waku network</br>
>></br>
>>The defaults are: -o "Topology.json"; -n 1; -t 1; -p 1; -T "configuration_model"</br>
> usage: $./generate_network --help
## batch_gen.sh
batch_gen.sh can generate given number of Waku networks and outputs them to a directory. Please make sure that the output directory does not exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
> usage: $./batch_gen.sh <output-dir> <#number of networks needed> </br>

View File

@ -19,18 +19,20 @@ getrand1(){
#n=$?
}
if [ "$#" -ne 2 ] || [ $2 -le 0 ] || ! [ -d "$1" ]; then
if [ "$#" -ne 2 ] || [ $2 -le 0 ] ; then
echo "usage: $0 <output dir> <#json files needed>" >&2
exit 1
fi
path=$1
nfiles=$2
mkdir -p $path
echo "Ok, will generate $nfiles networks & put them under '$path'."
prefix=$path"/WakuNet_"
suffix=".json"
nwtype="NEWMANWATTSSTROGATZ"
nodetype="DESKTOP"
for i in $(seq $nfiles)
do
@ -38,8 +40,8 @@ do
n=$((RANDOM+1))
getrand
t=$((RANDOM+1))
fname=$prefix$i$suffix
nwtype="configuration_model"
$(./generate_network.py -n $n -t $t -T $nwtype -o $fname)
echo "#$i\tn=$n\tt=$t\tT=$nwtype\to=$fname"
dirname="$path/$i/Waku"
mkdir "$path/$i"
echo "Generating ./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1 ...."
$(./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1)
done

View File

@ -4,79 +4,94 @@ import matplotlib.pyplot as plt
import networkx as nx
import random, math
import json
import argparse, os, sys
import sys, os
import string
import typer
from enum import Enum
# Enums & Consts
# To add a new node type, add appropriate entries to the nodeType and nodeTypeSwitch
class nodeType(Enum):
DESKTOP = "desktop" # waku desktop config
MOBILE = "mobile" # waku mobile config
nodeTypeSwitch = {
nodeType.DESKTOP : "rpc-admin = true\nkeep-alive = true\n",
nodeType.MOBILE : "rpc-admin = true\nkeep-alive = true\n"
}
# To add a new network type, add appropriate entries to the networkType and networkTypeSwitch
# the networkTypeSwitch is placed before generate_network(): fwd declaration mismatch with typer/python :/
class networkType(Enum):
CONFIGMODEL = "configmodel"
SCALEFREE = "scalefree" # power law
NEWMANWATTSSTROGATZ = "newmanwattsstrogatz" # mesh, smallworld
BARBELL = "barbell" # partition
BALANCEDTREE = "balancedtree" # committees?
STAR = "star" # spof
NW_DATA_FNAME = "network_data.json"
PREFIX = "waku_"
### I/O related fns ##############################################################
# Dump to a json file
def write_json(filename, data_2_dump):
json.dump(data_2_dump, open(filename,'w'), indent=2)
def write_json(dirname, json_dump):
fname = os.path.join(dirname, NW_DATA_FNAME)
with open(fname, "w") as f:
json.dump(json_dump, f, indent=2)
# has trouble with non-integer/non-hashable keys
def read_json(filename):
with open(filename) as f:
def write_toml(dirname, node_name, toml):
fname = os.path.join(dirname, f"{node_name}.toml")
with open(fname, "w") as f:
f.write(toml)
# Draw the network and output the image to a file
def draw(dirname, H):
nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
fname = os.path.join(dirname, NW_DATA_FNAME)
plt.savefig(f"{os.path.splitext(fname)[0]}.png", format="png")
plt.show()
# Has trouble with non-integer/non-hashable keys
def read_json(fname):
with open(fname) as f:
jdata = json.load(f)
return nx.node_link_graph(jdata)
# Draw the network and output the image to a file
def draw(fname, H):
nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
plt.savefig(os.path.splitext(fname)[0] + ".png", format="png")
plt.show()
def exists_and_nonempty(dirname):
if not os.path.exists(dirname):
return False
elif not os.path.isfile(dirname) and os.listdir(dirname):
print(f"{dirname}: exists and not empty")
return True
elif os.path.isfile(dirname):
print(f"{dirname}: exists but not a directory")
return True
else:
return False
# Initialize parser, set the defaults, and extract the options
def get_options():
parser = argparse.ArgumentParser(
prog = 'generate_network',
description = '''Generates and outputs
the Waku network conforming to input parameters''',
epilog = '''Defaults: -o "Topology.json";
-n 1; -t 1; -p 1; -T "configuration_model"
Supported nw types "configuration_model", "scalefree",
"newman_watts_strogatz"''')
parser.add_argument("-o", "--output",
default='Topology.json', dest='fname',
help='output json filename for the Waku network',
type=str, metavar='<file_name>')
parser.add_argument("-n", "--numnodes",
default=1, dest='num_nodes',
help='number of nodes in the Waku network',
type=int, metavar='<#nodes>')
parser.add_argument("-t", "--numtopics",
default=1, dest='num_topics',
help='number of topics in the Waku network',
type=int, metavar='<#topics>')
parser.add_argument("-T", "--type",
default="configuration_model", dest='nw_type',
help='network type of the Waku network',
type=str, metavar='<type>')
parser.add_argument("-p", "--numparts",
default=1, dest='num_partitions',
help='The number of partitions in the Waku network',
type=int, metavar='<#partitions>')
# parser.add_argument("-e", "--numedges",
# default=1, dest='num_edges',
# help='The number of edges in the Waku network',
# type=int, metavar='#edges>')
return parser.parse_args()
### topics related fns #############################################################
# Generate a random string of upper case chars
def generate_random_string(n):
return "".join(random.choice(string.ascii_uppercase) for _ in range(n))
# Generate a random string (UC chars) of len n
def generate_topic_string(n):
rs = ""
for _ in range(n):
r = random.randint(65, 65 + 26 - 1) # generate a random UC char
rs += (chr(r)) # append the char generated
return rs
# Generate the topics - UC chars prefixed by "topic"
# Generate the topics - topic followed by random UC chars - Eg, topic_XY"
def generate_topics(num_topics):
topics = []
base = 26
topic_len = int(math.log(num_topics)/math.log(base)) + 1
topics = {i: f"topic_{generate_topic_string(topic_len)}" for i in range(num_topics)}
topic_len = int(math.log(num_topics)/math.log(26)) + 1 # base is 26 - upper case letters
topics = {i: f"topic_{generate_random_string(topic_len)}" for i in range(num_topics)}
return topics
@ -91,12 +106,13 @@ def get_random_sublist(topics):
return sublist
### network processing related fns #################################################
# Network Types
# https://networkx.org/documentation/stable/reference/generated/networkx.generators.degree_seq.configuration_model.html
def generate_config_model(n):
#degrees = nx.random_powerlaw_tree_sequence(n, tries=10000)
degrees = [random.randint(1, n) for i in range(n)]
if (sum(degrees)) % 2 != 0: # adjust the degree to even
if (sum(degrees)) % 2 != 0: # adjust the degree to be even
degrees[-1] += 1
return nx.configuration_model(degrees) # generate the graph
@ -105,79 +121,90 @@ def generate_scalefree_graph(n):
return nx.scale_free_graph(n)
# n must be larger than k
def generate_newman_watts_strogatz_graph(n):
return nx.newman_watts_strogatz_graph(n, 12, 0.5)
# n must be larger than k=D=3
def generate_newmanwattsstrogatz_graph(n):
return nx.newman_watts_strogatz_graph(n, 3, 0.5)
def generate_barbell_graph(n):
return nx.barbell_graph(int(n/2), 1)
def generate_balanced_tree(n, fanout=3):
height = int(math.log(n)/math.log(fanout))
return nx.balanced_tree(fanout, height)
def generate_star_graph(n):
return nx.star_graph(n)
networkTypeSwitch = {
networkType.CONFIGMODEL : generate_config_model,
networkType.SCALEFREE : generate_scalefree_graph,
networkType.NEWMANWATTSSTROGATZ : generate_newmanwattsstrogatz_graph,
networkType.BARBELL : generate_barbell_graph,
networkType.BALANCEDTREE: generate_balanced_tree,
networkType.STAR : generate_star_graph
}
# Generate the network from nw type
def generate_network(num_nodes, nw_type, prefix):
G = nx.empty_graph()
if nw_type == "configuration_model":
G = generate_config_model(num_nodes)
elif nw_type == "scalefree":
G = generate_scalefree_graph(num_nodes)
elif nw_type == "newman_watts_strogatz":
G = generate_newman_watts_strogatz_graph(num_nodes)
else:
print(nw_type +": Unsupported network type")
sys.exit(1)
H = postprocess_network(G, prefix)
return H
def generate_network(n, nw_type):
return postprocess_network(networkTypeSwitch.get(nw_type)(n))
# used by generate_dump_data, *ought* to be global to handle partitions
ports_shifted = 0
def postprocess_network(G, prefix):
G = nx.Graph(G) # prune out parallel/multi edges
G.remove_edges_from(nx.selfloop_edges(G)) # Removing self-loops
# Labeling nodes to match waku containers
mapping = {i: f"{prefix}{i}" for i in range(len(G))}
return nx.relabel_nodes(G, mapping)
# Label the generated network with prefix
def postprocess_network(G):
G = nx.Graph(G) # prune out parallel/multi edges
G.remove_edges_from(nx.selfloop_edges(G)) # remove the self-loops
mapping = {i: f"{PREFIX}{i}" for i in range(len(G))}
return nx.relabel_nodes(G, mapping) # label the nodes
# Generate dump data from the network and topics
def generate_dump_data(H, topics):
data_to_dump = {}
global ports_shifted
### file format related fns ###########################################################
#Generate per node toml configs
def generate_toml(topics, node_type=nodeType.DESKTOP):
topic_str = " ".join(get_random_sublist(topics)) # space separated topics
return f"{nodeTypeSwitch.get(node_type)}topics = \"{topic_str}\"\n"
# Generates network-wide json and per-node toml and writes them
def generate_and_write_files(dirname, num_topics, H):
topics = generate_topics(num_topics)
json_dump = {}
for node in H.nodes:
data_to_dump[node] = {}
data_to_dump[node]["ports-shift"] = ports_shifted
ports_shifted += 1
data_to_dump[node]["topics"] = get_random_sublist(topics)
data_to_dump[node]["static-nodes"] = []
write_toml(dirname, node, generate_toml(topics)) # per node toml
json_dump[node] = {}
json_dump[node]["static-nodes"] = []
for edge in H.edges(node):
data_to_dump[node]["static-nodes"].append(edge[1])
return data_to_dump
json_dump[node]["static-nodes"].append(edge[1])
write_json(dirname, json_dump) # network wide json
def main():
#extract the CLI arguments and assign params
options = get_options()
fname = options.fname
num_nodes = options.num_nodes
num_topics = options.num_topics
nw_type = options.nw_type
prefix = "waku_"
num_partitions = options.num_partitions
#num_edges = options.num_edges ## need to control num_edges?
### the main ##########################################################################
def main(
dirname: str = "WakuNetwork", num_nodes: int = 4, num_topics: int = 1,
nw_type: networkType = networkType.NEWMANWATTSSTROGATZ.value,
node_type: nodeType = nodeType.DESKTOP.value,
num_partitions: int = 1):
if num_partitions > 1:
print("-p",num_partitions,
"Sorry, we do not yet support partitions")
print(f"--num-partitions {num_partitions}, Sorry, we do not yet support partitions")
sys.exit(1)
# Generate the network and postprocess it
H = generate_network(num_nodes, nw_type, prefix)
# Generate the topics
topics = generate_topics(num_topics)
# Generate the dump data
dump_data = generate_dump_data(H, topics)
# Dump the network in a json file
write_json(fname, dump_data)
# Display the graph
draw(fname, H)
# Generate the network
G = generate_network(num_nodes, nw_type)
# Refuse to overwrite non-empty dirs
if exists_and_nonempty(dirname) :
sys.exit(1)
os.makedirs(dirname, exist_ok=True)
# Generate file format specific data structs and write the files; optionally, draw the network
generate_and_write_files(dirname, num_topics, G)
draw(dirname, G)
if __name__ == "__main__":
main()
typer.run(main)

View File

@ -1,41 +0,0 @@
IMAGE_NAME = "statusteam/nim-waku:deploy-status-prod"
# Waku RPC Port
RPC_PORT_ID = "rpc"
RPC_TCP_PORT = 8545
# Waku Matrics Port
PROMETHEUS_PORT_ID = "prometheus"
PROMETHEUS_TCP_PORT = 8008
GET_WAKU_INFO_METHOD = "get_waku_v2_debug_v1_info"
CONNECT_TO_PEER_METHOD = "post_waku_v2_admin_v1_peers"
def run(args):
# in case u want to run each json separately, follow this cmd line arg format for main.star
#kurtosis run . --args '{"json_nw_name": "github.com/user/kurto-module/json_dir/abc.json"}'
json_loc=args.json_nw_name
file_contents = read_file(json_loc)
#print(file_contents)
decoded = json.decode(file_contents)
services ={}
# Get up all waku nodes
for wakunode_name in decoded.keys():
waku_service = add_service(
service_id=wakunode_name,
config=struct(
image=IMAGE_NAME,
ports={
RPC_PORT_ID: struct(number=RPC_TCP_PORT, protocol="TCP"),
PROMETHEUS_PORT_ID: struct(number=PROMETHEUS_TCP_PORT, protocol="TCP")
},
entrypoint=[
"/usr/bin/wakunode", "--rpc-address=0.0.0.0", "--metrics-server-address=0.0.0.0"
],
cmd=[
"--topics='" + " ".join(decoded[wakunode_name]["topics"]) + "'", '--rpc-admin=true', '--keep-alive=true', '--metrics-server=true',
]
)
)
services[wakunode_name] = waku_service

View File

@ -1,35 +0,0 @@
#!/bin/sh
# -> symlink - ln -s source/dir .
# step 0)
# symlink this script and the main.star to the root of ur kurtosis module.
#
# step 1)
# put the json files you want to run kurtosis in a directory
#
# step 2)
# copy that entire directory to the root of your kurtosis module
# !!! WARNING: symlinking the directory will NOT work !!!
#
# step 3)
# run this script in the kurtosis module root dir of ur module
if [ "$#" -ne 2 ] || ! [ -d "$1" ]; then
echo "usage: $0 <input dir> <repo prefix>" >&2
exit 1
fi
path=$1
repo=$2
echo "Ok, will run kurtosis on all .json networks under '$path'."
for json in "$path"/*.json
do
cmd="kurtosis run . --args '{\"json_nw_name\": \"$repo/$json\"}'"
echo $cmd
eval $cmd
done
echo $repo, $path, "DONE!"