diff --git a/.gitignore b/.gitignore index ae2de8a..cd7b9e2 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,10 @@ *.swo *~ +# dirs +Waku +WakuNetwork + # local json Topology.* topology.* diff --git a/Readme.md b/Readme.md index a1de36e..2e90cc3 100644 --- a/Readme.md +++ b/Readme.md @@ -1,45 +1,12 @@ -This repo contains the scripts to generate different network models for wakukurtosis runs. +This repo contains scripts to generate network models (in JSON) and waku configuration files (in TOMLs) for wakukurtosis runs. -## run_kurtosis_tests.sh -run_kurtosis_tests.sh will run kurtosis on a set of json files in a directory. It requires two arguments. First is a directory containing json files; other file types in the directory are ignored. Second is the github root/prefix of the kurtosis module you run the tests under.
- -> usage: ./run_kurtosis_tests.sh
- -Running this script is somewhat complicated; so follow the following instructions to a dot. You **WILL** require the main.star provided here. The main.star reads a input json and instantiates Waku nodes accordingly. The runs are repeated for each of the input json files under the specified directory. - -#### step 0) - symlink run_kurtosis_tests.sh to the root directory of your kurtosis module.
-#### step 1) - backup the your kurtosis module's own main.star. copy the main.star provided here to the root directory of your kurtosis module.
- !!! WARNING: symlinking the main.star will NOT work !!!
-#### step 3) - put all the json files you want to use in a directory. Call it *Foo*
-#### step 3) - copy the *Foo* directory to the root of your kurtosis module
- !!! WARNING: symlinking the directory will NOT work !!!
-#### step 4) - run this script in the root directory of the kurtosis module. provide the directory (*Foo*) and the github root/prefix of the kurtosis module as arguments to the script
- - -## gen_jsons.sh -gen_jsons.sh can generate given number of Waku networs and outputs them to a directory. Please make sure that the output directory exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.
- -> usage: ./gen_jsons.sh <#json files needed>
## generate_network.py -generate_network.py can generate networks with specified number of nodes and topics. the network types currently supported is "configuration_model" and more are on the way. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph. +generate_network.py generates one network and per-node configuration files. The tool is configurable with specified number of nodes, topics, network types, node types. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph. -> usage: generate_network [-h] [-o ] [-n <#nodes>] [-t <#topics>] - [-T ]
->>
->> Generates and outputs the Waku network conforming to input parameters ->>
->> optional arguments:
->>   -h, --help show this help message and exit
->>   -o , --output output json filename for the Waku network
->>   -n <#nodes>, --numnodes <#nodes> number of nodes in the Waku network
->>   -t <#topics>, --numtopics <#topics> number of topics in the Waku network
->>   -T , --type network type for the Waku network
->>   -p <#partitions>, --numparts <#partitions> number of partitions in the Waku network
- >>
->>The defaults are: -o "Topology.json"; -n 1; -t 1; -p 1; -T "configuration_model"
+> usage: $./generate_network --help + +## batch_gen.sh +batch_gen.sh can generate given number of Waku networks and outputs them to a directory. Please make sure that the output directory does not exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.
+ +> usage: $./batch_gen.sh <#number of networks needed>
diff --git a/gen_jsons.sh b/batch_gen.sh similarity index 58% rename from gen_jsons.sh rename to batch_gen.sh index 15e1042..3ea571b 100755 --- a/gen_jsons.sh +++ b/batch_gen.sh @@ -19,18 +19,20 @@ getrand1(){ #n=$? } -if [ "$#" -ne 2 ] || [ $2 -le 0 ] || ! [ -d "$1" ]; then +if [ "$#" -ne 2 ] || [ $2 -le 0 ] ; then echo "usage: $0 <#json files needed>" >&2 exit 1 fi path=$1 nfiles=$2 +mkdir -p $path echo "Ok, will generate $nfiles networks & put them under '$path'." + +nwtype="NEWMANWATTSSTROGATZ" +nodetype="DESKTOP" -prefix=$path"/WakuNet_" -suffix=".json" for i in $(seq $nfiles) do @@ -38,8 +40,8 @@ do n=$((RANDOM+1)) getrand t=$((RANDOM+1)) - fname=$prefix$i$suffix - nwtype="configuration_model" - $(./generate_network.py -n $n -t $t -T $nwtype -o $fname) - echo "#$i\tn=$n\tt=$t\tT=$nwtype\to=$fname" + dirname="$path/$i/Waku" + mkdir "$path/$i" + echo "Generating ./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1 ...." + $(./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1) done diff --git a/generate_network.py b/generate_network.py index 67a0896..c40045d 100755 --- a/generate_network.py +++ b/generate_network.py @@ -4,79 +4,94 @@ import matplotlib.pyplot as plt import networkx as nx import random, math import json -import argparse, os, sys +import sys, os +import string +import typer +from enum import Enum + + +# Enums & Consts + +# To add a new node type, add appropriate entries to the nodeType and nodeTypeSwitch +class nodeType(Enum): + DESKTOP = "desktop" # waku desktop config + MOBILE = "mobile" # waku mobile config + +nodeTypeSwitch = { + nodeType.DESKTOP : "rpc-admin = true\nkeep-alive = true\n", + nodeType.MOBILE : "rpc-admin = true\nkeep-alive = true\n" + } + + +# To add a new network type, add appropriate entries to the networkType and networkTypeSwitch +# the networkTypeSwitch is placed before generate_network(): fwd declaration mismatch with typer/python :/ +class networkType(Enum): + CONFIGMODEL = "configmodel" + SCALEFREE = "scalefree" # power law + NEWMANWATTSSTROGATZ = "newmanwattsstrogatz" # mesh, smallworld + BARBELL = "barbell" # partition + BALANCEDTREE = "balancedtree" # committees? + STAR = "star" # spof + + +NW_DATA_FNAME = "network_data.json" +PREFIX = "waku_" + + +### I/O related fns ############################################################## # Dump to a json file -def write_json(filename, data_2_dump): - json.dump(data_2_dump, open(filename,'w'), indent=2) +def write_json(dirname, json_dump): + fname = os.path.join(dirname, NW_DATA_FNAME) + with open(fname, "w") as f: + json.dump(json_dump, f, indent=2) -# has trouble with non-integer/non-hashable keys -def read_json(filename): - with open(filename) as f: +def write_toml(dirname, node_name, toml): + fname = os.path.join(dirname, f"{node_name}.toml") + with open(fname, "w") as f: + f.write(toml) + + +# Draw the network and output the image to a file +def draw(dirname, H): + nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True) + fname = os.path.join(dirname, NW_DATA_FNAME) + plt.savefig(f"{os.path.splitext(fname)[0]}.png", format="png") + plt.show() + + +# Has trouble with non-integer/non-hashable keys +def read_json(fname): + with open(fname) as f: jdata = json.load(f) return nx.node_link_graph(jdata) -# Draw the network and output the image to a file -def draw(fname, H): - nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True) - plt.savefig(os.path.splitext(fname)[0] + ".png", format="png") - plt.show() +def exists_and_nonempty(dirname): + if not os.path.exists(dirname): + return False + elif not os.path.isfile(dirname) and os.listdir(dirname): + print(f"{dirname}: exists and not empty") + return True + elif os.path.isfile(dirname): + print(f"{dirname}: exists but not a directory") + return True + else: + return False -# Initialize parser, set the defaults, and extract the options -def get_options(): - parser = argparse.ArgumentParser( - prog = 'generate_network', - description = '''Generates and outputs - the Waku network conforming to input parameters''', - epilog = '''Defaults: -o "Topology.json"; - -n 1; -t 1; -p 1; -T "configuration_model" - Supported nw types "configuration_model", "scalefree", - "newman_watts_strogatz"''') - parser.add_argument("-o", "--output", - default='Topology.json', dest='fname', - help='output json filename for the Waku network', - type=str, metavar='') - parser.add_argument("-n", "--numnodes", - default=1, dest='num_nodes', - help='number of nodes in the Waku network', - type=int, metavar='<#nodes>') - parser.add_argument("-t", "--numtopics", - default=1, dest='num_topics', - help='number of topics in the Waku network', - type=int, metavar='<#topics>') - parser.add_argument("-T", "--type", - default="configuration_model", dest='nw_type', - help='network type of the Waku network', - type=str, metavar='') - parser.add_argument("-p", "--numparts", - default=1, dest='num_partitions', - help='The number of partitions in the Waku network', - type=int, metavar='<#partitions>') -# parser.add_argument("-e", "--numedges", -# default=1, dest='num_edges', -# help='The number of edges in the Waku network', -# type=int, metavar='#edges>') - return parser.parse_args() +### topics related fns ############################################################# + +# Generate a random string of upper case chars +def generate_random_string(n): + return "".join(random.choice(string.ascii_uppercase) for _ in range(n)) -# Generate a random string (UC chars) of len n -def generate_topic_string(n): - rs = "" - for _ in range(n): - r = random.randint(65, 65 + 26 - 1) # generate a random UC char - rs += (chr(r)) # append the char generated - return rs - - -# Generate the topics - UC chars prefixed by "topic" +# Generate the topics - topic followed by random UC chars - Eg, topic_XY" def generate_topics(num_topics): - topics = [] - base = 26 - topic_len = int(math.log(num_topics)/math.log(base)) + 1 - topics = {i: f"topic_{generate_topic_string(topic_len)}" for i in range(num_topics)} + topic_len = int(math.log(num_topics)/math.log(26)) + 1 # base is 26 - upper case letters + topics = {i: f"topic_{generate_random_string(topic_len)}" for i in range(num_topics)} return topics @@ -91,12 +106,13 @@ def get_random_sublist(topics): return sublist +### network processing related fns ################################################# + # Network Types -# https://networkx.org/documentation/stable/reference/generated/networkx.generators.degree_seq.configuration_model.html def generate_config_model(n): #degrees = nx.random_powerlaw_tree_sequence(n, tries=10000) degrees = [random.randint(1, n) for i in range(n)] - if (sum(degrees)) % 2 != 0: # adjust the degree to even + if (sum(degrees)) % 2 != 0: # adjust the degree to be even degrees[-1] += 1 return nx.configuration_model(degrees) # generate the graph @@ -105,79 +121,90 @@ def generate_scalefree_graph(n): return nx.scale_free_graph(n) -# n must be larger than k -def generate_newman_watts_strogatz_graph(n): - return nx.newman_watts_strogatz_graph(n, 12, 0.5) +# n must be larger than k=D=3 +def generate_newmanwattsstrogatz_graph(n): + return nx.newman_watts_strogatz_graph(n, 3, 0.5) + + +def generate_barbell_graph(n): + return nx.barbell_graph(int(n/2), 1) + + +def generate_balanced_tree(n, fanout=3): + height = int(math.log(n)/math.log(fanout)) + return nx.balanced_tree(fanout, height) + + +def generate_star_graph(n): + return nx.star_graph(n) + + +networkTypeSwitch = { + networkType.CONFIGMODEL : generate_config_model, + networkType.SCALEFREE : generate_scalefree_graph, + networkType.NEWMANWATTSSTROGATZ : generate_newmanwattsstrogatz_graph, + networkType.BARBELL : generate_barbell_graph, + networkType.BALANCEDTREE: generate_balanced_tree, + networkType.STAR : generate_star_graph + } # Generate the network from nw type -def generate_network(num_nodes, nw_type, prefix): - G = nx.empty_graph() - if nw_type == "configuration_model": - G = generate_config_model(num_nodes) - elif nw_type == "scalefree": - G = generate_scalefree_graph(num_nodes) - elif nw_type == "newman_watts_strogatz": - G = generate_newman_watts_strogatz_graph(num_nodes) - else: - print(nw_type +": Unsupported network type") - sys.exit(1) - H = postprocess_network(G, prefix) - return H +def generate_network(n, nw_type): + return postprocess_network(networkTypeSwitch.get(nw_type)(n)) -# used by generate_dump_data, *ought* to be global to handle partitions -ports_shifted = 0 -def postprocess_network(G, prefix): - G = nx.Graph(G) # prune out parallel/multi edges - G.remove_edges_from(nx.selfloop_edges(G)) # Removing self-loops - # Labeling nodes to match waku containers - mapping = {i: f"{prefix}{i}" for i in range(len(G))} - return nx.relabel_nodes(G, mapping) +# Label the generated network with prefix +def postprocess_network(G): + G = nx.Graph(G) # prune out parallel/multi edges + G.remove_edges_from(nx.selfloop_edges(G)) # remove the self-loops + mapping = {i: f"{PREFIX}{i}" for i in range(len(G))} + return nx.relabel_nodes(G, mapping) # label the nodes -# Generate dump data from the network and topics -def generate_dump_data(H, topics): - data_to_dump = {} - global ports_shifted +### file format related fns ########################################################### +#Generate per node toml configs +def generate_toml(topics, node_type=nodeType.DESKTOP): + topic_str = " ".join(get_random_sublist(topics)) # space separated topics + return f"{nodeTypeSwitch.get(node_type)}topics = \"{topic_str}\"\n" + + +# Generates network-wide json and per-node toml and writes them +def generate_and_write_files(dirname, num_topics, H): + topics = generate_topics(num_topics) + json_dump = {} for node in H.nodes: - data_to_dump[node] = {} - data_to_dump[node]["ports-shift"] = ports_shifted - ports_shifted += 1 - data_to_dump[node]["topics"] = get_random_sublist(topics) - data_to_dump[node]["static-nodes"] = [] + write_toml(dirname, node, generate_toml(topics)) # per node toml + json_dump[node] = {} + json_dump[node]["static-nodes"] = [] for edge in H.edges(node): - data_to_dump[node]["static-nodes"].append(edge[1]) - return data_to_dump + json_dump[node]["static-nodes"].append(edge[1]) + write_json(dirname, json_dump) # network wide json -def main(): - #extract the CLI arguments and assign params - options = get_options() - fname = options.fname - num_nodes = options.num_nodes - num_topics = options.num_topics - nw_type = options.nw_type - prefix = "waku_" - num_partitions = options.num_partitions - #num_edges = options.num_edges ## need to control num_edges? +### the main ########################################################################## +def main( + dirname: str = "WakuNetwork", num_nodes: int = 4, num_topics: int = 1, + nw_type: networkType = networkType.NEWMANWATTSSTROGATZ.value, + node_type: nodeType = nodeType.DESKTOP.value, + num_partitions: int = 1): if num_partitions > 1: - print("-p",num_partitions, - "Sorry, we do not yet support partitions") + print(f"--num-partitions {num_partitions}, Sorry, we do not yet support partitions") sys.exit(1) - # Generate the network and postprocess it - H = generate_network(num_nodes, nw_type, prefix) - # Generate the topics - topics = generate_topics(num_topics) - # Generate the dump data - dump_data = generate_dump_data(H, topics) - # Dump the network in a json file - write_json(fname, dump_data) - # Display the graph - draw(fname, H) + # Generate the network + G = generate_network(num_nodes, nw_type) + + # Refuse to overwrite non-empty dirs + if exists_and_nonempty(dirname) : + sys.exit(1) + os.makedirs(dirname, exist_ok=True) + + # Generate file format specific data structs and write the files; optionally, draw the network + generate_and_write_files(dirname, num_topics, G) + draw(dirname, G) if __name__ == "__main__": - main() + typer.run(main) diff --git a/main.star b/main.star deleted file mode 100644 index 34ec34d..0000000 --- a/main.star +++ /dev/null @@ -1,41 +0,0 @@ -IMAGE_NAME = "statusteam/nim-waku:deploy-status-prod" - -# Waku RPC Port -RPC_PORT_ID = "rpc" -RPC_TCP_PORT = 8545 - -# Waku Matrics Port -PROMETHEUS_PORT_ID = "prometheus" -PROMETHEUS_TCP_PORT = 8008 - -GET_WAKU_INFO_METHOD = "get_waku_v2_debug_v1_info" -CONNECT_TO_PEER_METHOD = "post_waku_v2_admin_v1_peers" - -def run(args): - # in case u want to run each json separately, follow this cmd line arg format for main.star - #kurtosis run . --args '{"json_nw_name": "github.com/user/kurto-module/json_dir/abc.json"}' - json_loc=args.json_nw_name - file_contents = read_file(json_loc) - #print(file_contents) - decoded = json.decode(file_contents) - services ={} - - # Get up all waku nodes - for wakunode_name in decoded.keys(): - waku_service = add_service( - service_id=wakunode_name, - config=struct( - image=IMAGE_NAME, - ports={ - RPC_PORT_ID: struct(number=RPC_TCP_PORT, protocol="TCP"), - PROMETHEUS_PORT_ID: struct(number=PROMETHEUS_TCP_PORT, protocol="TCP") - }, - entrypoint=[ - "/usr/bin/wakunode", "--rpc-address=0.0.0.0", "--metrics-server-address=0.0.0.0" - ], - cmd=[ - "--topics='" + " ".join(decoded[wakunode_name]["topics"]) + "'", '--rpc-admin=true', '--keep-alive=true', '--metrics-server=true', - ] - ) - ) - services[wakunode_name] = waku_service diff --git a/run_kurtosis_tests.sh b/run_kurtosis_tests.sh deleted file mode 100755 index 3d316ac..0000000 --- a/run_kurtosis_tests.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/bin/sh - -# -> symlink - ln -s source/dir . - -# step 0) -# symlink this script and the main.star to the root of ur kurtosis module. -# -# step 1) -# put the json files you want to run kurtosis in a directory -# -# step 2) -# copy that entire directory to the root of your kurtosis module -# !!! WARNING: symlinking the directory will NOT work !!! -# -# step 3) -# run this script in the kurtosis module root dir of ur module - - -if [ "$#" -ne 2 ] || ! [ -d "$1" ]; then - echo "usage: $0 " >&2 - exit 1 -fi - -path=$1 -repo=$2 -echo "Ok, will run kurtosis on all .json networks under '$path'." - -for json in "$path"/*.json -do - cmd="kurtosis run . --args '{\"json_nw_name\": \"$repo/$json\"}'" - echo $cmd - eval $cmd -done - -echo $repo, $path, "DONE!"