Added typer, combined JSON/TOMLs generations; added network/node types; refactored (#4)

* typer, json/toml, more network/node type, refactor

* cleanup

* PR#4 mods

* typer + enums + switch

* gitignore & minor edits

.gitignore

@@ -3,6 +3,10 @@
 *.swo
 *~
 
+# dirs
+Waku
+WakuNetwork
+
 # local json
 Topology.*
 topology.*

Readme.md

@@ -1,45 +1,12 @@
-This repo contains the scripts to generate different network models for wakukurtosis runs. 
+This repo contains scripts to generate network models (in JSON) and waku configuration files (in TOMLs) for wakukurtosis runs. 
 
-## run_kurtosis_tests.sh
-run_kurtosis_tests.sh will run kurtosis on a set of json files in a directory. It requires two arguments. First is a directory containing json files; other file types in the directory are ignored. Second is the github root/prefix of the kurtosis module you run the tests under.</br>
-
-> usage: ./run_kurtosis_tests.sh <input_dir> <repo_prefix> </br>
-
-Running this script is somewhat complicated; so follow the following instructions to a dot. You **WILL** require the main.star provided here. The main.star reads a input json and instantiates Waku nodes accordingly. The runs are repeated for each of the input json files under the specified directory.
-
-#### step 0)
-  symlink  run_kurtosis_tests.sh to the root directory of your kurtosis module.</br>
-#### step 1)
-  backup the your kurtosis module's own main.star. copy the main.star provided here to the root directory of your kurtosis module.</br>
-     !!! WARNING: symlinking the main.star will NOT work !!!</br>
-#### step 3)
-  put all the json files you want to use in a directory. Call it *Foo*</br>
-#### step 3)
-   copy the *Foo* directory to the root of your kurtosis module</br>
-   !!! WARNING: symlinking the directory will NOT work !!!</br>
-#### step 4)
-   run this script in the root directory of the kurtosis module. provide the directory (*Foo*) and the github root/prefix of the kurtosis module as arguments to the script</br>
-
-
-## gen_jsons.sh
-gen_jsons.sh can generate given number of Waku networs and outputs them to a directory. Please make sure that the output directory exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
-
-> usage: ./gen_jsons.sh <output_dir> <#json files needed> </br>
 
 ## generate_network.py
-generate_network.py can generate networks with specified number of nodes and topics. the network types currently supported is "configuration_model" and more are on the way. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
+generate_network.py generates one network and per-node configuration files. The tool is configurable with specified number of nodes, topics, network types, node types. Use with Python3. Comment out the `#draw(fname, H)` line to visualise the generated graph.
 
-> usage: generate_network [-h] [-o <file_name>] [-n <#nodes>] [-t <#topics>]
+> usage: $./generate_network --help
-                        [-T <type>] <br>
+
->> </br>
+## batch_gen.sh
->> Generates and outputs the Waku network conforming to input parameters<//br>
+batch_gen.sh can generate given number of Waku networks and outputs them to a directory. Please make sure that the output directory does not exists; both relative and absolute paths work. The Wakunode parameters are generated at random; edit the MIN and MAX for finer control. The script requires bc & /dev/urandom.<br>
->> </br>
+
->> optional arguments:</br>
+> usage: $./batch_gen.sh <output-dir> <#number of networks needed> </br>
->> &emsp;  -h, --help            show this help message and exit</br>
->> &emsp;  -o <file_name>, --output <file_name> output json filename for the Waku network </br>
->> &emsp;  -n <#nodes>, --numnodes <#nodes> number of nodes in the Waku network </br>
->> &emsp;  -t <#topics>, --numtopics <#topics> number of topics in the Waku network </br>
->> &emsp;  -T <type>, --type <type>  network type for the Waku network </br>
->> &emsp;  -p <#partitions>, --numparts <#partitions> number of partitions in the Waku network</br>
-  >></br>
->>The defaults are: -o "Topology.json"; -n 1; -t 1; -p 1; -T "configuration_model"</br>

batch_gen.sh

@@ -19,18 +19,20 @@ getrand1(){
   #n=$? 
 }
 
-if [ "$#" -ne 2 ] || [ $2 -le 0 ] || ! [ -d "$1" ]; then
+if [ "$#" -ne 2 ] || [ $2 -le 0 ] ; then
   echo "usage: $0 <output dir> <#json files needed>" >&2
   exit 1
 fi
 
 path=$1
 nfiles=$2
+mkdir -p $path
 
 echo "Ok, will generate $nfiles networks & put them  under '$path'."
+ 
+nwtype="NEWMANWATTSSTROGATZ"
+nodetype="DESKTOP"
 
-prefix=$path"/WakuNet_"
-suffix=".json"
 
 for i in $(seq $nfiles)
 do
@@ -38,8 +40,8 @@ do
   n=$((RANDOM+1))
   getrand 
   t=$((RANDOM+1))
-  fname=$prefix$i$suffix
+  dirname="$path/$i/Waku"
-  nwtype="configuration_model"
+  mkdir "$path/$i"
-  $(./generate_network.py -n $n -t $t -T $nwtype -o $fname)
+  echo "Generating ./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1 ...."
-  echo "#$i\tn=$n\tt=$t\tT=$nwtype\to=$fname"
+  $(./generate_network.py --dirname $dirname --num-nodes $n --num-topics $t --nw-type $nwtype --node-type $nodetype --num-partitions 1)
 done

generate_network.py

@@ -4,79 +4,94 @@ import matplotlib.pyplot as plt
 import networkx as nx
 import random, math
 import json
-import argparse, os, sys
+import sys, os
+import string
+import typer
+from enum import Enum
+
+
+# Enums & Consts
+
+# To add a new node type, add appropriate entries to the nodeType and nodeTypeSwitch
+class nodeType(Enum):
+    DESKTOP = "desktop" # waku desktop config
+    MOBILE  = "mobile"  # waku mobile config
+
+nodeTypeSwitch = {
+        nodeType.DESKTOP : "rpc-admin = true\nkeep-alive = true\n",
+        nodeType.MOBILE  : "rpc-admin = true\nkeep-alive = true\n"
+    }
+
+
+# To add a new network type, add appropriate entries to the networkType and networkTypeSwitch
+# the networkTypeSwitch is placed before generate_network(): fwd declaration mismatch with typer/python :/
+class networkType(Enum):
+    CONFIGMODEL         = "configmodel"
+    SCALEFREE           = "scalefree"           # power law
+    NEWMANWATTSSTROGATZ = "newmanwattsstrogatz" # mesh, smallworld
+    BARBELL             = "barbell"             # partition
+    BALANCEDTREE        = "balancedtree"        # committees?
+    STAR                = "star"                # spof
+
+
+NW_DATA_FNAME = "network_data.json"
+PREFIX = "waku_"
+
+
+### I/O related fns ##############################################################
 
 # Dump to a json file
-def write_json(filename, data_2_dump):
+def write_json(dirname, json_dump):
-    json.dump(data_2_dump, open(filename,'w'), indent=2)
+    fname = os.path.join(dirname, NW_DATA_FNAME)
+    with open(fname, "w") as f:
+        json.dump(json_dump, f, indent=2)
 
 
-# has trouble with non-integer/non-hashable keys 
+def write_toml(dirname, node_name, toml):
-def read_json(filename):
+    fname = os.path.join(dirname, f"{node_name}.toml")
-    with open(filename) as f:
+    with open(fname, "w") as f:
+        f.write(toml)
+
+
+# Draw the network and output the image to a file
+def draw(dirname, H):
+    nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
+    fname = os.path.join(dirname, NW_DATA_FNAME)
+    plt.savefig(f"{os.path.splitext(fname)[0]}.png", format="png")
+    plt.show()
+
+
+# Has trouble with non-integer/non-hashable keys 
+def read_json(fname):
+    with open(fname) as f:
         jdata = json.load(f)
     return nx.node_link_graph(jdata)
 
 
-# Draw the network and output the image to a file
+def exists_and_nonempty(dirname):
-def draw(fname, H):
+    if not os.path.exists(dirname):
-    nx.draw(H, pos=nx.kamada_kawai_layout(H), with_labels=True)
+        return False
-    plt.savefig(os.path.splitext(fname)[0] + ".png", format="png")
+    elif not os.path.isfile(dirname) and os.listdir(dirname):
-    plt.show()
+        print(f"{dirname}: exists and not empty")
+        return True
+    elif os.path.isfile(dirname):
+        print(f"{dirname}: exists but not a directory")
+        return True
+    else:
+        return False
 
 
-# Initialize parser, set the defaults, and extract the options
+### topics related fns #############################################################
-def get_options():
+
-    parser = argparse.ArgumentParser(
+# Generate a random string of upper case chars
-            prog = 'generate_network',
+def generate_random_string(n):
-            description = '''Generates and outputs 
+    return "".join(random.choice(string.ascii_uppercase) for _ in range(n))
-            the Waku network conforming to input parameters''',
-            epilog = '''Defaults: -o "Topology.json"; 
-            -n 1; -t 1; -p 1; -T "configuration_model"
-            Supported nw types "configuration_model", "scalefree",
-            "newman_watts_strogatz"''')
-    parser.add_argument("-o", "--output", 
-            default='Topology.json', dest='fname', 
-            help='output json filename for the Waku network', 
-            type=str, metavar='<file_name>')
-    parser.add_argument("-n", "--numnodes", 
-            default=1, dest='num_nodes', 
-            help='number of nodes in the Waku network', 
-            type=int, metavar='<#nodes>')
-    parser.add_argument("-t", "--numtopics", 
-            default=1, dest='num_topics', 
-            help='number of topics in the Waku network', 
-            type=int, metavar='<#topics>')
-    parser.add_argument("-T", "--type", 
-            default="configuration_model", dest='nw_type', 
-            help='network type of the Waku network', 
-            type=str, metavar='<type>')
-    parser.add_argument("-p", "--numparts", 
-            default=1, dest='num_partitions', 
-            help='The number of partitions in the Waku network', 
-            type=int, metavar='<#partitions>')
-#    parser.add_argument("-e", "--numedges",
-#            default=1, dest='num_edges', 
-#            help='The number of edges in the Waku network', 
-#            type=int, metavar='#edges>')
-    return parser.parse_args()
 
 
-# Generate a random string (UC chars) of len n
+# Generate the topics - topic followed by random UC chars - Eg, topic_XY"
-def generate_topic_string(n):
-    rs = ""
-    for _ in range(n):
-        r = random.randint(65, 65 + 26 - 1) # generate a random UC char
-        rs += (chr(r))                      # append the char generated
-    return rs
-
-
-# Generate the topics - UC chars prefixed by "topic"
 def generate_topics(num_topics):
-    topics = []
+    topic_len = int(math.log(num_topics)/math.log(26)) + 1  # base is 26 - upper case letters
-    base = 26
+    topics = {i: f"topic_{generate_random_string(topic_len)}" for i in range(num_topics)}
-    topic_len = int(math.log(num_topics)/math.log(base)) + 1
-    topics = {i: f"topic_{generate_topic_string(topic_len)}" for i in range(num_topics)}
     return topics
 
 
@@ -91,12 +106,13 @@ def get_random_sublist(topics):
     return sublist
 
 
+### network processing related fns #################################################
+
 # Network Types
-# https://networkx.org/documentation/stable/reference/generated/networkx.generators.degree_seq.configuration_model.html
 def generate_config_model(n):
     #degrees = nx.random_powerlaw_tree_sequence(n, tries=10000)
     degrees = [random.randint(1, n) for i in range(n)]
-    if (sum(degrees)) % 2 != 0:         # adjust the degree to even
+    if (sum(degrees)) % 2 != 0:         # adjust the degree to be even
         degrees[-1] += 1
     return nx.configuration_model(degrees) # generate the graph
 
@@ -105,79 +121,90 @@ def generate_scalefree_graph(n):
     return  nx.scale_free_graph(n)
 
 
-# n must be larger than k
+# n must be larger than k=D=3
-def generate_newman_watts_strogatz_graph(n):
+def generate_newmanwattsstrogatz_graph(n):
-    return nx.newman_watts_strogatz_graph(n, 12, 0.5)
+    return nx.newman_watts_strogatz_graph(n, 3, 0.5)
+
+
+def generate_barbell_graph(n):
+    return nx.barbell_graph(int(n/2), 1)
+
+
+def generate_balanced_tree(n, fanout=3):
+    height = int(math.log(n)/math.log(fanout))
+    return nx.balanced_tree(fanout, height)
+
+
+def generate_star_graph(n):
+    return nx.star_graph(n)
+
+
+networkTypeSwitch = {
+        networkType.CONFIGMODEL : generate_config_model,
+        networkType.SCALEFREE   : generate_scalefree_graph,
+        networkType.NEWMANWATTSSTROGATZ : generate_newmanwattsstrogatz_graph,
+        networkType.BARBELL     : generate_barbell_graph,
+        networkType.BALANCEDTREE: generate_balanced_tree,
+        networkType.STAR        : generate_star_graph
+    }
 
 
 # Generate the network from nw type
-def generate_network(num_nodes, nw_type, prefix):
+def generate_network(n, nw_type):
-    G = nx.empty_graph()
+       return postprocess_network(networkTypeSwitch.get(nw_type)(n))
-    if nw_type == "configuration_model":
-        G = generate_config_model(num_nodes)
-    elif nw_type == "scalefree":
-        G = generate_scalefree_graph(num_nodes)
-    elif nw_type == "newman_watts_strogatz":
-        G = generate_newman_watts_strogatz_graph(num_nodes) 
-    else: 
-        print(nw_type +": Unsupported network type")
-        sys.exit(1)
-    H = postprocess_network(G, prefix)
-    return H
 
 
-# used by generate_dump_data, *ought* to be global to handle partitions
+# Label the generated network with prefix
-ports_shifted = 0
+def postprocess_network(G):
-def postprocess_network(G, prefix):
+    G = nx.Graph(G)                             # prune out parallel/multi edges
-    G = nx.Graph(G)         # prune out parallel/multi edges
+    G.remove_edges_from(nx.selfloop_edges(G))   # remove the self-loops
-    G.remove_edges_from(nx.selfloop_edges(G))   # Removing self-loops
+    mapping = {i: f"{PREFIX}{i}" for i in range(len(G))}
-    # Labeling nodes to match waku containers
+    return nx.relabel_nodes(G, mapping)         # label the nodes
-    mapping = {i: f"{prefix}{i}" for i in range(len(G))}
-    return nx.relabel_nodes(G, mapping)
 
 
-# Generate dump data from the network and topics
+### file format related fns ###########################################################
-def generate_dump_data(H, topics):
+#Generate per node toml configs
-    data_to_dump = {}
+def generate_toml(topics, node_type=nodeType.DESKTOP):
-    global ports_shifted
+    topic_str = " ".join(get_random_sublist(topics))   # space separated topics
+    return f"{nodeTypeSwitch.get(node_type)}topics = \"{topic_str}\"\n"
+
+
+# Generates network-wide json and per-node toml and writes them 
+def generate_and_write_files(dirname, num_topics, H):
+    topics = generate_topics(num_topics)
+    json_dump = {}
     for node in H.nodes:
-        data_to_dump[node] = {}
+        write_toml(dirname, node, generate_toml(topics))        # per node toml
-        data_to_dump[node]["ports-shift"] = ports_shifted
+        json_dump[node] = {}
-        ports_shifted += 1
+        json_dump[node]["static-nodes"] = []
-        data_to_dump[node]["topics"] = get_random_sublist(topics)
-        data_to_dump[node]["static-nodes"] = []
         for edge in H.edges(node):
-            data_to_dump[node]["static-nodes"].append(edge[1])
+            json_dump[node]["static-nodes"].append(edge[1])
-    return data_to_dump
+    write_json(dirname, json_dump)                              # network wide json
 
 
-def main():
+### the main ##########################################################################
-    #extract the CLI arguments and assign params
+def main(
-    options = get_options()
+        dirname: str = "WakuNetwork", num_nodes: int = 4, num_topics: int = 1, 
-    fname = options.fname
+        nw_type: networkType = networkType.NEWMANWATTSSTROGATZ.value, 
-    num_nodes = options.num_nodes
+        node_type: nodeType = nodeType.DESKTOP.value,
-    num_topics = options.num_topics
+        num_partitions: int = 1):
-    nw_type = options.nw_type
-    prefix = "waku_"     
-    num_partitions = options.num_partitions
-    #num_edges = options.num_edges    ## need to control num_edges?
 
     if num_partitions > 1:
-        print("-p",num_partitions,
+        print(f"--num-partitions {num_partitions}, Sorry, we do not yet support partitions")
-                "Sorry, we do not yet support partitions")
         sys.exit(1)
 
-    # Generate the network and postprocess it
+    # Generate the network
-    H = generate_network(num_nodes, nw_type, prefix)
+    G = generate_network(num_nodes, nw_type)
-    # Generate the topics
+
-    topics = generate_topics(num_topics)
+    # Refuse to overwrite non-empty dirs
-    # Generate the dump data
+    if exists_and_nonempty(dirname) :
-    dump_data = generate_dump_data(H, topics)
+        sys.exit(1)
-    # Dump the network in a json file
+    os.makedirs(dirname, exist_ok=True)
-    write_json(fname, dump_data)
+
-    # Display the graph
+    # Generate file format specific data structs and write the files; optionally, draw the network
-    draw(fname, H)
+    generate_and_write_files(dirname, num_topics, G)
+    draw(dirname, G)
 
 
 if __name__ == "__main__":
-    main()
+    typer.run(main)

main.star

@@ -1,41 +0,0 @@
-IMAGE_NAME = "statusteam/nim-waku:deploy-status-prod"
-
-# Waku RPC Port
-RPC_PORT_ID = "rpc"
-RPC_TCP_PORT = 8545
-
-# Waku Matrics Port
-PROMETHEUS_PORT_ID = "prometheus"
-PROMETHEUS_TCP_PORT = 8008
-
-GET_WAKU_INFO_METHOD = "get_waku_v2_debug_v1_info"
-CONNECT_TO_PEER_METHOD = "post_waku_v2_admin_v1_peers"
-
-def run(args):
-    # in case u want to run each json separately, follow this cmd line arg format for main.star 
-    #kurtosis run . --args '{"json_nw_name": "github.com/user/kurto-module/json_dir/abc.json"}'
-    json_loc=args.json_nw_name
-    file_contents = read_file(json_loc)
-    #print(file_contents)
-    decoded = json.decode(file_contents)
-    services ={}
-
-    # Get up all waku nodes
-    for wakunode_name in decoded.keys():
-        waku_service = add_service(
-            service_id=wakunode_name,
-            config=struct(
-                image=IMAGE_NAME,
-                ports={ 
-                    RPC_PORT_ID: struct(number=RPC_TCP_PORT, protocol="TCP"),
-                    PROMETHEUS_PORT_ID: struct(number=PROMETHEUS_TCP_PORT, protocol="TCP")
-                    },
-                entrypoint=[
-                    "/usr/bin/wakunode", "--rpc-address=0.0.0.0", "--metrics-server-address=0.0.0.0"
-                ],
-                cmd=[
-                    "--topics='" + " ".join(decoded[wakunode_name]["topics"]) + "'", '--rpc-admin=true', '--keep-alive=true',  '--metrics-server=true',
-                ]
-            )
-        )
-        services[wakunode_name] = waku_service

run_kurtosis_tests.sh

@@ -1,35 +0,0 @@
-#!/bin/sh
-
-# -> symlink - ln -s source/dir .
-
-# step 0)
-#  symlink this script and the main.star to the root of ur kurtosis module.
-#
-# step 1)
-#  put the json files you want to run kurtosis in a directory
-#
-# step 2)
-#   copy that entire directory to the root of your kurtosis module
-#   !!! WARNING: symlinking the directory will NOT work !!!
-#
-# step 3)
-#   run this script in the kurtosis module root dir of ur module
-
-
-if [ "$#" -ne 2 ] || ! [ -d "$1" ]; then
-  echo "usage: $0 <input dir> <repo prefix>" >&2
-  exit 1
-fi
-
-path=$1
-repo=$2
-echo "Ok, will run kurtosis on all .json networks under '$path'."
-
-for json in "$path"/*.json
-do
- cmd="kurtosis run . --args '{\"json_nw_name\": \"$repo/$json\"}'"
- echo $cmd
- eval $cmd
-done
-
-echo $repo, $path, "DONE!"