Added typer, combined JSON/TOMLs generations; added network/node types; refactored (#4)
* typer, json/toml, more network/node type, refactor * cleanup * PR#4 mods * typer + enums + switch * gitignore & minor edits
This commit is contained in:
0xFugue
GitHub
parent
d677a4c78d
commit
87433027be
|
|
@ -3,6 +3,10 @@
|
|||
*.swo
|
||||
*~
|
||||
|
||||
# dirs
|
||||
Waku
|
||||
WakuNetwork
|
||||
|
||||
# local json
|
||||
Topology.*
|
||||
topology.*
|
||||
|
|
|
|||
49
Readme.md
49
Readme.md
|
|
@ -1,45 +1,12 @@
|
|||
This repo contains the scripts to generate different network models for wakukurtosis runs.
|
||||
This repo contains scripts to generate network models (in JSON) and waku configuration files (in TOMLs) for wakukurtosis runs.
|
||||
|
||||
## run_kurtosis_tests.sh
|
||||
run_kurtosis_tests.sh will run kurtosis on a set of json files in a directory. It requires two arguments. First is a directory containing json files; other file types in the directory are ignored. Second is the github root/prefix of the kurtosis module you run the tests under.</br>
|
||||
|
||||
> usage: ./run_kurtosis_tests.sh <input_dir> <repo_prefix> </br>
|
||||
|
||||
Running this script is somewhat complicated; so follow the following instructions to a dot. You **WILL** require the main.star provided here. The main.star reads a input json and instantiates Waku nodes accordingly. The runs are repeated for each of the input json files under the specified directory.
|
||||
|
||||
#### step 0)
|
||||
symlink run_kurtosis_tests.sh to the root directory of your kurtosis module.</br>
|
||||
#### step 1)
|
||||
backup the your kurtosis module's own main.star. copy the main.star provided here to the root directory of your kurtosis module.</br>
|
||||
!!! WARNING: symlinking the main.star will NOT work !!!</br>
|
||||
#### step 3)
|
||||
put all the json files you want to use in a directory. Call it *Foo*</br>
|
||||
#### step 3)
|
||||
copy the *Foo* directory to the root of your kurtosis module</br>
|
||||
!!! WARNING: symlinking the directory will NOT work !!!</br>
|
||||
#### step 4)
|
||||
run this script in the root directory of the kurtosis module. provide the directory (*Foo*) and the github root/prefix of the kurtosis module as arguments to the script</br>
|
||||
|
||||
|
||||
## gen_jsons.sh
|
||||
gen_jsons.sh can generate given number of Waku networs and outputs them to a directory. Please make sure that the output directory exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
|
||||
|
||||
> usage: ./gen_jsons.sh <output_dir> <#json files needed> </br>
|
||||
|
||||
## generate_network.py
|
||||
generate_network.py can generate networks with specified number of nodes and topics. the network types currently supported is "configuration_model" and more are on the way. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
|
||||
generate_network.py generates one network and per-node configuration files. The tool is configurable with specified number of nodes, topics, network types, node types. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
|
||||
|
||||
> usage: generate_network [-h] [-o <file_name>] [-n <#nodes>] [-t <#topics>]
|
||||
[-T <type>] <br>
|
||||
>> </br>
|
||||
>> Generates and outputs the Waku network conforming to input parameters<//br>
|
||||
>> </br>
|
||||
>> optional arguments:</br>
|
||||
>>   -h, --help show this help message and exit</br>
|
||||
>>   -o <file_name>, --output <file_name> output json filename for the Waku network </br>
|
||||
>>   -n <#nodes>, --numnodes <#nodes> number of nodes in the Waku network </br>
|
||||
>>   -t <#topics>, --numtopics <#topics> number of topics in the Waku network </br>
|
||||
>>   -T <type>, --type <type> network type for the Waku network </br>
|
||||
>>   -p <#partitions>, --numparts <#partitions> number of partitions in the Waku network</br>
|
||||
>></br>
|
||||
>>The defaults are: -o "Topology.json"; -n 1; -t 1; -p 1; -T "configuration_model"</br>
|
||||
> usage: $./generate_network --help
|
||||
|
||||
## batch_gen.sh
|
||||
batch_gen.sh can generate given number of Waku networks and outputs them to a directory. Please make sure that the output directory does not exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
|
||||
|
||||
> usage: $./batch_gen.sh <output-dir> <#number of networks needed> </br>
|
||||
|
|
|
|||
|
|
@ -19,18 +19,20 @@ getrand1(){
|
|||
#n=$?
|
||||
}
|
||||
|
||||
if [ "$#" -ne 2 ] || [ $2 -le 0 ] || ! [ -d "$1" ]; then
|
||||
if [ "$#" -ne 2 ] || [ $2 -le 0 ] ; then
|
||||
echo "usage: $0 <output dir> <#json files needed>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
path=$1
|
||||
nfiles=$2
|
||||
mkdir -p $path
|
||||
|
||||
echo "Ok, will generate $nfiles networks & put them under '$path'."
|
||||
|
||||
nwtype="NEWMANWATTSSTROGATZ"
|
||||
nodetype="DESKTOP"
|
||||
|
||||
prefix=$path"/WakuNet_"
|
||||
suffix=".json"
|
||||
|
||||
for i in $(seq $nfiles)
|
||||
do
|
||||
|
|
@ -38,8 +40,8 @@ do
|
|||
n=$((RANDOM+1))
|
||||
getrand
|
||||
t=$((RANDOM+1))
|
||||
fname=$prefix$i$suffix
|
||||
nwtype="configuration_model"
|
||||
$(./generate_network.py -n $n -t $t -T $nwtype -o $fname)
|
||||
echo "#$i\tn=$n\tt=$t\tT=$nwtype\to=$fname"
|
||||
dirname="$path/$i/Waku"
|
||||
mkdir "$path/$i"
|
||||
echo "Generating ./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1 ...."
|
||||
$(./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1)
|
||||
done
|
||||
|
|
@ -4,79 +4,94 @@ import matplotlib.pyplot as plt
|
|||
import networkx as nx
|
||||
import random, math
|
||||
import json
|
||||
import argparse, os, sys
|
||||
import sys, os
|
||||
import string
|
||||
import typer
|
||||
from enum import Enum
|
||||
|
||||
|
||||
# Enums & Consts
|
||||
|
||||
# To add a new node type, add appropriate entries to the nodeType and nodeTypeSwitch
|
||||
class nodeType(Enum):
|
||||
DESKTOP = "desktop" # waku desktop config
|
||||
MOBILE = "mobile" # waku mobile config
|
||||
|
||||
nodeTypeSwitch = {
|
||||
nodeType.DESKTOP : "rpc-admin = true\nkeep-alive = true\n",
|
||||
nodeType.MOBILE : "rpc-admin = true\nkeep-alive = true\n"
|
||||
}
|
||||
|
||||
|
||||
# To add a new network type, add appropriate entries to the networkType and networkTypeSwitch
|
||||
# the networkTypeSwitch is placed before generate_network(): fwd declaration mismatch with typer/python :/
|
||||
class networkType(Enum):
|
||||
CONFIGMODEL = "configmodel"
|
||||
SCALEFREE = "scalefree" # power law
|
||||
NEWMANWATTSSTROGATZ = "newmanwattsstrogatz" # mesh, smallworld
|
||||
BARBELL = "barbell" # partition
|
||||
BALANCEDTREE = "balancedtree" # committees?
|
||||
STAR = "star" # spof
|
||||
|
||||
|
||||
NW_DATA_FNAME = "network_data.json"
|
||||
PREFIX = "waku_"
|
||||
|
||||
|
||||
### I/O related fns ##############################################################
|
||||
|
||||
# Dump to a json file
|
||||
def write_json(filename, data_2_dump):
|
||||
json.dump(data_2_dump, open(filename,'w'), indent=2)
|
||||
def write_json(dirname, json_dump):
|
||||
fname = os.path.join(dirname, NW_DATA_FNAME)
|
||||
with open(fname, "w") as f:
|
||||
json.dump(json_dump, f, indent=2)
|
||||
|
||||
|
||||
# has trouble with non-integer/non-hashable keys
|
||||
def read_json(filename):
|
||||
with open(filename) as f:
|
||||
def write_toml(dirname, node_name, toml):
|
||||
fname = os.path.join(dirname, f"{node_name}.toml")
|
||||
with open(fname, "w") as f:
|
||||
f.write(toml)
|
||||
|
||||
|
||||
# Draw the network and output the image to a file
|
||||
def draw(dirname, H):
|
||||
nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
|
||||
fname = os.path.join(dirname, NW_DATA_FNAME)
|
||||
plt.savefig(f"{os.path.splitext(fname)[0]}.png", format="png")
|
||||
plt.show()
|
||||
|
||||
|
||||
# Has trouble with non-integer/non-hashable keys
|
||||
def read_json(fname):
|
||||
with open(fname) as f:
|
||||
jdata = json.load(f)
|
||||
return nx.node_link_graph(jdata)
|
||||
|
||||
|
||||
# Draw the network and output the image to a file
|
||||
def draw(fname, H):
|
||||
nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
|
||||
plt.savefig(os.path.splitext(fname)[0] + ".png", format="png")
|
||||
plt.show()
|
||||
def exists_and_nonempty(dirname):
|
||||
if not os.path.exists(dirname):
|
||||
return False
|
||||
elif not os.path.isfile(dirname) and os.listdir(dirname):
|
||||
print(f"{dirname}: exists and not empty")
|
||||
return True
|
||||
elif os.path.isfile(dirname):
|
||||
print(f"{dirname}: exists but not a directory")
|
||||
return True
|
||||
else:
|
||||
return False
|
||||
|
||||
|
||||
# Initialize parser, set the defaults, and extract the options
|
||||
def get_options():
|
||||
parser = argparse.ArgumentParser(
|
||||
prog = 'generate_network',
|
||||
description = '''Generates and outputs
|
||||
the Waku network conforming to input parameters''',
|
||||
epilog = '''Defaults: -o "Topology.json";
|
||||
-n 1; -t 1; -p 1; -T "configuration_model"
|
||||
Supported nw types "configuration_model", "scalefree",
|
||||
"newman_watts_strogatz"''')
|
||||
parser.add_argument("-o", "--output",
|
||||
default='Topology.json', dest='fname',
|
||||
help='output json filename for the Waku network',
|
||||
type=str, metavar='<file_name>')
|
||||
parser.add_argument("-n", "--numnodes",
|
||||
default=1, dest='num_nodes',
|
||||
help='number of nodes in the Waku network',
|
||||
type=int, metavar='<#nodes>')
|
||||
parser.add_argument("-t", "--numtopics",
|
||||
default=1, dest='num_topics',
|
||||
help='number of topics in the Waku network',
|
||||
type=int, metavar='<#topics>')
|
||||
parser.add_argument("-T", "--type",
|
||||
default="configuration_model", dest='nw_type',
|
||||
help='network type of the Waku network',
|
||||
type=str, metavar='<type>')
|
||||
parser.add_argument("-p", "--numparts",
|
||||
default=1, dest='num_partitions',
|
||||
help='The number of partitions in the Waku network',
|
||||
type=int, metavar='<#partitions>')
|
||||
# parser.add_argument("-e", "--numedges",
|
||||
# default=1, dest='num_edges',
|
||||
# help='The number of edges in the Waku network',
|
||||
# type=int, metavar='#edges>')
|
||||
return parser.parse_args()
|
||||
### topics related fns #############################################################
|
||||
|
||||
# Generate a random string of upper case chars
|
||||
def generate_random_string(n):
|
||||
return "".join(random.choice(string.ascii_uppercase) for _ in range(n))
|
||||
|
||||
|
||||
# Generate a random string (UC chars) of len n
|
||||
def generate_topic_string(n):
|
||||
rs = ""
|
||||
for _ in range(n):
|
||||
r = random.randint(65, 65 + 26 - 1) # generate a random UC char
|
||||
rs += (chr(r)) # append the char generated
|
||||
return rs
|
||||
|
||||
|
||||
# Generate the topics - UC chars prefixed by "topic"
|
||||
# Generate the topics - topic followed by random UC chars - Eg, topic_XY"
|
||||
def generate_topics(num_topics):
|
||||
topics = []
|
||||
base = 26
|
||||
topic_len = int(math.log(num_topics)/math.log(base)) + 1
|
||||
topics = {i: f"topic_{generate_topic_string(topic_len)}" for i in range(num_topics)}
|
||||
topic_len = int(math.log(num_topics)/math.log(26)) + 1 # base is 26 - upper case letters
|
||||
topics = {i: f"topic_{generate_random_string(topic_len)}" for i in range(num_topics)}
|
||||
return topics
|
||||
|
||||
|
||||
|
|
@ -91,12 +106,13 @@ def get_random_sublist(topics):
|
|||
return sublist
|
||||
|
||||
|
||||
### network processing related fns #################################################
|
||||
|
||||
# Network Types
|
||||
# https://networkx.org/documentation/stable/reference/generated/networkx.generators.degree_seq.configuration_model.html
|
||||
def generate_config_model(n):
|
||||
#degrees = nx.random_powerlaw_tree_sequence(n, tries=10000)
|
||||
degrees = [random.randint(1, n) for i in range(n)]
|
||||
if (sum(degrees)) % 2 != 0: # adjust the degree to even
|
||||
if (sum(degrees)) % 2 != 0: # adjust the degree to be even
|
||||
degrees[-1] += 1
|
||||
return nx.configuration_model(degrees) # generate the graph
|
||||
|
||||
|
|
@ -105,79 +121,90 @@ def generate_scalefree_graph(n):
|
|||
return nx.scale_free_graph(n)
|
||||
|
||||
|
||||
# n must be larger than k
|
||||
def generate_newman_watts_strogatz_graph(n):
|
||||
return nx.newman_watts_strogatz_graph(n, 12, 0.5)
|
||||
# n must be larger than k=D=3
|
||||
def generate_newmanwattsstrogatz_graph(n):
|
||||
return nx.newman_watts_strogatz_graph(n, 3, 0.5)
|
||||
|
||||
|
||||
def generate_barbell_graph(n):
|
||||
return nx.barbell_graph(int(n/2), 1)
|
||||
|
||||
|
||||
def generate_balanced_tree(n, fanout=3):
|
||||
height = int(math.log(n)/math.log(fanout))
|
||||
return nx.balanced_tree(fanout, height)
|
||||
|
||||
|
||||
def generate_star_graph(n):
|
||||
return nx.star_graph(n)
|
||||
|
||||
|
||||
networkTypeSwitch = {
|
||||
networkType.CONFIGMODEL : generate_config_model,
|
||||
networkType.SCALEFREE : generate_scalefree_graph,
|
||||
networkType.NEWMANWATTSSTROGATZ : generate_newmanwattsstrogatz_graph,
|
||||
networkType.BARBELL : generate_barbell_graph,
|
||||
networkType.BALANCEDTREE: generate_balanced_tree,
|
||||
networkType.STAR : generate_star_graph
|
||||
}
|
||||
|
||||
|
||||
# Generate the network from nw type
|
||||
def generate_network(num_nodes, nw_type, prefix):
|
||||
G = nx.empty_graph()
|
||||
if nw_type == "configuration_model":
|
||||
G = generate_config_model(num_nodes)
|
||||
elif nw_type == "scalefree":
|
||||
G = generate_scalefree_graph(num_nodes)
|
||||
elif nw_type == "newman_watts_strogatz":
|
||||
G = generate_newman_watts_strogatz_graph(num_nodes)
|
||||
else:
|
||||
print(nw_type +": Unsupported network type")
|
||||
sys.exit(1)
|
||||
H = postprocess_network(G, prefix)
|
||||
return H
|
||||
def generate_network(n, nw_type):
|
||||
return postprocess_network(networkTypeSwitch.get(nw_type)(n))
|
||||
|
||||
|
||||
# used by generate_dump_data, *ought* to be global to handle partitions
|
||||
ports_shifted = 0
|
||||
def postprocess_network(G, prefix):
|
||||
G = nx.Graph(G) # prune out parallel/multi edges
|
||||
G.remove_edges_from(nx.selfloop_edges(G)) # Removing self-loops
|
||||
# Labeling nodes to match waku containers
|
||||
mapping = {i: f"{prefix}{i}" for i in range(len(G))}
|
||||
return nx.relabel_nodes(G, mapping)
|
||||
# Label the generated network with prefix
|
||||
def postprocess_network(G):
|
||||
G = nx.Graph(G) # prune out parallel/multi edges
|
||||
G.remove_edges_from(nx.selfloop_edges(G)) # remove the self-loops
|
||||
mapping = {i: f"{PREFIX}{i}" for i in range(len(G))}
|
||||
return nx.relabel_nodes(G, mapping) # label the nodes
|
||||
|
||||
|
||||
# Generate dump data from the network and topics
|
||||
def generate_dump_data(H, topics):
|
||||
data_to_dump = {}
|
||||
global ports_shifted
|
||||
### file format related fns ###########################################################
|
||||
#Generate per node toml configs
|
||||
def generate_toml(topics, node_type=nodeType.DESKTOP):
|
||||
topic_str = " ".join(get_random_sublist(topics)) # space separated topics
|
||||
return f"{nodeTypeSwitch.get(node_type)}topics = \"{topic_str}\"\n"
|
||||
|
||||
|
||||
# Generates network-wide json and per-node toml and writes them
|
||||
def generate_and_write_files(dirname, num_topics, H):
|
||||
topics = generate_topics(num_topics)
|
||||
json_dump = {}
|
||||
for node in H.nodes:
|
||||
data_to_dump[node] = {}
|
||||
data_to_dump[node]["ports-shift"] = ports_shifted
|
||||
ports_shifted += 1
|
||||
data_to_dump[node]["topics"] = get_random_sublist(topics)
|
||||
data_to_dump[node]["static-nodes"] = []
|
||||
write_toml(dirname, node, generate_toml(topics)) # per node toml
|
||||
json_dump[node] = {}
|
||||
json_dump[node]["static-nodes"] = []
|
||||
for edge in H.edges(node):
|
||||
data_to_dump[node]["static-nodes"].append(edge[1])
|
||||
return data_to_dump
|
||||
json_dump[node]["static-nodes"].append(edge[1])
|
||||
write_json(dirname, json_dump) # network wide json
|
||||
|
||||
|
||||
def main():
|
||||
#extract the CLI arguments and assign params
|
||||
options = get_options()
|
||||
fname = options.fname
|
||||
num_nodes = options.num_nodes
|
||||
num_topics = options.num_topics
|
||||
nw_type = options.nw_type
|
||||
prefix = "waku_"
|
||||
num_partitions = options.num_partitions
|
||||
#num_edges = options.num_edges ## need to control num_edges?
|
||||
### the main ##########################################################################
|
||||
def main(
|
||||
dirname: str = "WakuNetwork", num_nodes: int = 4, num_topics: int = 1,
|
||||
nw_type: networkType = networkType.NEWMANWATTSSTROGATZ.value,
|
||||
node_type: nodeType = nodeType.DESKTOP.value,
|
||||
num_partitions: int = 1):
|
||||
|
||||
if num_partitions > 1:
|
||||
print("-p",num_partitions,
|
||||
"Sorry, we do not yet support partitions")
|
||||
print(f"--num-partitions {num_partitions}, Sorry, we do not yet support partitions")
|
||||
sys.exit(1)
|
||||
|
||||
# Generate the network and postprocess it
|
||||
H = generate_network(num_nodes, nw_type, prefix)
|
||||
# Generate the topics
|
||||
topics = generate_topics(num_topics)
|
||||
# Generate the dump data
|
||||
dump_data = generate_dump_data(H, topics)
|
||||
# Dump the network in a json file
|
||||
write_json(fname, dump_data)
|
||||
# Display the graph
|
||||
draw(fname, H)
|
||||
# Generate the network
|
||||
G = generate_network(num_nodes, nw_type)
|
||||
|
||||
# Refuse to overwrite non-empty dirs
|
||||
if exists_and_nonempty(dirname) :
|
||||
sys.exit(1)
|
||||
os.makedirs(dirname, exist_ok=True)
|
||||
|
||||
# Generate file format specific data structs and write the files; optionally, draw the network
|
||||
generate_and_write_files(dirname, num_topics, G)
|
||||
draw(dirname, G)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
typer.run(main)
|
||||
|
|
|
|||
41
main.star
41
main.star
|
|
@ -1,41 +0,0 @@
|
|||
IMAGE_NAME = "statusteam/nim-waku:deploy-status-prod"
|
||||
|
||||
# Waku RPC Port
|
||||
RPC_PORT_ID = "rpc"
|
||||
RPC_TCP_PORT = 8545
|
||||
|
||||
# Waku Matrics Port
|
||||
PROMETHEUS_PORT_ID = "prometheus"
|
||||
PROMETHEUS_TCP_PORT = 8008
|
||||
|
||||
GET_WAKU_INFO_METHOD = "get_waku_v2_debug_v1_info"
|
||||
CONNECT_TO_PEER_METHOD = "post_waku_v2_admin_v1_peers"
|
||||
|
||||
def run(args):
|
||||
# in case u want to run each json separately, follow this cmd line arg format for main.star
|
||||
#kurtosis run . --args '{"json_nw_name": "github.com/user/kurto-module/json_dir/abc.json"}'
|
||||
json_loc=args.json_nw_name
|
||||
file_contents = read_file(json_loc)
|
||||
#print(file_contents)
|
||||
decoded = json.decode(file_contents)
|
||||
services ={}
|
||||
|
||||
# Get up all waku nodes
|
||||
for wakunode_name in decoded.keys():
|
||||
waku_service = add_service(
|
||||
service_id=wakunode_name,
|
||||
config=struct(
|
||||
image=IMAGE_NAME,
|
||||
ports={
|
||||
RPC_PORT_ID: struct(number=RPC_TCP_PORT, protocol="TCP"),
|
||||
PROMETHEUS_PORT_ID: struct(number=PROMETHEUS_TCP_PORT, protocol="TCP")
|
||||
},
|
||||
entrypoint=[
|
||||
"/usr/bin/wakunode", "--rpc-address=0.0.0.0", "--metrics-server-address=0.0.0.0"
|
||||
],
|
||||
cmd=[
|
||||
"--topics='" + " ".join(decoded[wakunode_name]["topics"]) + "'", '--rpc-admin=true', '--keep-alive=true', '--metrics-server=true',
|
||||
]
|
||||
)
|
||||
)
|
||||
services[wakunode_name] = waku_service
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
#!/bin/sh
|
||||
|
||||
# -> symlink - ln -s source/dir .
|
||||
|
||||
# step 0)
|
||||
# symlink this script and the main.star to the root of ur kurtosis module.
|
||||
#
|
||||
# step 1)
|
||||
# put the json files you want to run kurtosis in a directory
|
||||
#
|
||||
# step 2)
|
||||
# copy that entire directory to the root of your kurtosis module
|
||||
# !!! WARNING: symlinking the directory will NOT work !!!
|
||||
#
|
||||
# step 3)
|
||||
# run this script in the kurtosis module root dir of ur module
|
||||
|
||||
|
||||
if [ "$#" -ne 2 ] || ! [ -d "$1" ]; then
|
||||
echo "usage: $0 <input dir> <repo prefix>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
path=$1
|
||||
repo=$2
|
||||
echo "Ok, will run kurtosis on all .json networks under '$path'."
|
||||
|
||||
for json in "$path"/*.json
|
||||
do
|
||||
cmd="kurtosis run . --args '{\"json_nw_name\": \"$repo/$json\"}'"
|
||||
echo $cmd
|
||||
eval $cmd
|
||||
done
|
||||
|
||||
echo $repo, $path, "DONE!"
|
||||
Loading…
Reference in New Issue