beacon node sim: various improvements (fixes #111) (#156)

* allow running more or fewer validators
* use deterministic key generation for tests to avoid exhausting system
RNG
* update README with simulator docs
* write the data of each validator to separate file, instead of a big
chainstart.json (makes it easier to run different validator counts)

README.md

@@ -5,9 +5,13 @@
 [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
 ![Stability: experimental](https://img.shields.io/badge/stability-experimental-orange.svg)
 
-An alternative implementation of the Ethereum beacon chain in Nim.
+Nimbus beacon chain is a research implementation of the beacon chain component of the upcoming Ethereum Serenity upgrade, aka eth2. See the main [Nimbus](https://github.com/status-im/nimbus/) project for the bigger picture.
 
-Please see [Full Beacon chain](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md) specs and the Ethereum Foundation [reference implementation](https://github.com/ethereum/beacon_chain).
+## Related
+
+* [status-im/nimbus](https://github.com/status-im/nimbus/): main Nimbus repository - start here to learn more about the Nimbus eco-system
+* [ethereum/eth2.0-specs](https://github.com/ethereum/eth2.0-specs/blob/master/specs/core/0_beacon-chain.md): Serenity specification that this project implements
+* [ethereum/beacon_chain]((https://github.com/ethereum/beacon_chain): reference implementation from the Ethereum foundation
 
 You can check where the beacon chain fits in the Ethereum research ecosystem in the [Status Athenaeum](https://github.com/status-im/athenaeum/blob/b465626cc551e361492e56d32517b2cdadd7493f/ethereum_research_records.json#L38).
 
@@ -15,6 +19,73 @@ You can check where the beacon chain fits in the Ethereum research ecosystem in
 
 The Eth 2.0 test vectors and their generators are available in a [dedicated repository](https://github.com/status-im/eth2-testgen).
 
+## Building and Testing
+
+The beacon chain components require that you have Nim installed - the easiest way to get started is to head over to the main [Nimbus](https://github.com/status-im/nimbus/) repository and follow the build instructions there. Once you're able to build nimbus:
+
+Then:
+
+```bash
+# Or wherever you cloned nimbus
+cd nimbus
+
+# Prep environment
+make
+./env.sh bash
+
+# You're now in a shell environment that has the right Nim version available
+
+cd..
+
+git clone https://github.com/status-im/nim-beacon-chain
+cd nim-beacon-chain
+
+# Build binaries and run test suite
+nimble build
+nimble test
+```
+
+This should produce some passing tests.
+
+## Beacon node simulation
+
+The beacon node simulation is will create a full peer-to-peer network of beacon nodes and validators, and run the beacon chain in real time. To change network parameters such as shard and validator counts, see [start.sh](tests/simulation/start.sh).
+
+```bash
+# Start beacon chain simulation, resuming from the previous state (if any)
+./tests/simulation/start.sh
+
+# Clear data from last run and restart simulation with a new genesis block
+rf -rf tests/simulation/data ; ./tests/simulation/start.sh
+
+# Run an extra node - by default the network will launch with 9 nodes, each
+# hosting 10 validators. The last 10 validators are lazy bums that hid from the
+# startup script, but you can command them back to work in a separate terminal
+# with:
+./tests/simulation/run_node.sh 9
+```
+
+You can also separate the output from each beacon node in its own panel, using [multitail](http://www.vanheusden.com/multitail/):
+
+```bash
+USE_MULTITAIL="yes" ./tests/simulation/start.sh
+```
+
+You can find out more about it in the [development update](https://our.status.im/nimbus-development-update-2018-12-2/).
+
+_Alternatively, fire up our [experimental Vagrant instance with Nim pre-installed](https://our.status.im/setting-up-a-local-vagrant-environment-for-nim-development/) and give us yout feedback about the process!_
+
+## State transition simulation
+
+The state transition simulator can quickly run the Beacon chain state transition function in isolation and output JSON snapshots of the state. The simulation runs without networking and blocks are processed without slot time delays.
+
+```bash
+cd research
+# build and run state simulator, then display its help - -d:release speeds it
+# up substantially, allowing the simulation of longer runs in reasonable time
+nim c -d:release -r state_sim --help
+```
+
 ## Convention
 
 Ethereum Foundation uses:
@@ -29,44 +100,6 @@ Nim NEP-1 recommends:
 
 To facilitate collaboration and comparison, Nim-beacon-chain uses the Ethereum Foundation convention.
 
-## Installation
-
-You can install the developement version of the library through nimble with the following command
-```
-nimble install https://github.com/status-im/nim-beacon-chain@#master
-```
-
-## Building and Testing
-
-To try out the implementation, please first make sure you have a [Nim environment configured](https://bitfalls.com/2018/10/09/introduction-into-the-nim-language/).
-
-_Alternatively, fire up our [experimental Vagrant instance with Nim pre-installed](https://our.status.im/setting-up-a-local-vagrant-environment-for-nim-development/) and give us yout feedback about the process!_
-
-Then:
-
-```bash
-git clone https://github.com/status-im/nim-beacon-chain
-cd nim-beacon-chain
-nimble install
-nimble test
-```
-
-This should produce some passing tests.
-
-Additionally, you can run our simulation which generates a genesis file from some randomly generated validators. It then fires up 10 beacon nodes (each hosting 9 validators) which talk to each other and try to do state transitions. The simulation can be run by executing:
-
-```bash
-bash tests/simulation/start.sh
-```
-
-You can also separate the output from each beacon node in its own panel, using [multitail](http://www.vanheusden.com/multitail/):
-
-```bash
-USE_MULTITAIL="yes" ./tests/simulation/start.sh
-```
-
-You can find out more about it in the [development update](https://our.status.im/nimbus-development-update-2018-12-2/).
-
 ## License
 
 Licensed and distributed under either of

beacon_chain/beacon_node.nim

@@ -177,9 +177,8 @@ template findIt(s: openarray, predicate: untyped): int =
   res
 
 proc addLocalValidators*(node: BeaconNode) =
-  for validator in node.config.validators:
+  for privKey in node.config.validators:
     let
-      privKey = validator.privKey
       pubKey = privKey.pubKey()
 
     let idx = node.state.data.validator_registry.findIt(it.pubKey == pubKey)
@@ -276,7 +275,7 @@ proc makeAttestation(node: BeaconNode,
       beacon_block_root: node.state.blck.root,
       epoch_boundary_root: Eth2Digest(), # TODO
       shard_block_root: Eth2Digest(), # TODO
-      latest_crosslink: Crosslink(epoch: state.latest_crosslinks[shard].epoch),
+      latest_crosslink: state.latest_crosslinks[shard],
       justified_epoch: state.justified_epoch,
       justified_block_root: justifiedBlockRoot)
 
@@ -632,8 +631,8 @@ when isMainModule:
   case config.cmd
   of createChain:
     createStateSnapshot(
-      config.chainStartupData, config.genesisOffset,
+      config.validatorsDir.string, config.numValidators, config.firstValidator,
-      config.outputStateFile.string)
+      config.genesisOffset, config.outputStateFile.string)
     quit 0
 
   of noCommand:

beacon_chain/conf.nim

@@ -14,12 +14,6 @@ type
     noCommand
     createChain
 
-  ChainStartupData* = object
-    validatorDeposits*: seq[Deposit]
-
-  PrivateValidatorData* = object
-    privKey*: ValidatorPrivKey
-
   BeaconNodeConf* = object
     logLevel* {.
       desc: "Sets the log level",
@@ -54,24 +48,32 @@ type
 
       validators* {.
         required
-        desc: "A path to a pair of public and private keys for a validator. " &
+        desc: "Path to a validator private key, as generated by validator_keygen"
-              "Nimbus will automatically add the extensions .privkey and .pubkey."
         longform: "validator"
-        shortform: "v".}: seq[PrivateValidatorData]
+        shortform: "v".}: seq[ValidatorPrivKey]
 
       stateSnapshot* {.
         desc: "Json file specifying a recent state snapshot"
         shortform: "s".}: Option[BeaconState]
 
     of createChain:
-      chainStartupData* {.
+      validatorsDir* {.
+        desc: "Directory containing validator descriptors named vXXXXXXX.deposit.json"
+        shortform: "d".}: DirPath
+
+      numValidators* {.
         desc: ""
-        shortform: "c".}: ChainStartupData
+        shortform: "n".}: int
+
+      firstValidator* {.
+        desc: "index of first validator to add to validator list"
+        shortform: "o"
+        defaultValue: 0.}: int
 
       genesisOffset* {.
         desc: "Seconds from now to add to genesis time"
         shortForm: "g"
-        defaultValue: 0 .}: int
+        defaultValue: 5 .}: int
 
       outputStateFile* {.
         desc: "Output file where to write the initial state snapshot"
@@ -98,5 +100,5 @@ template handledAsJsonFilename(T: untyped) {.dirty.} =
     return Json.loadFile(string(input), T)
 
 handledAsJsonFilename BeaconState
-handledAsJsonFilename ChainStartupData
+handledAsJsonFilename Deposit
-handledAsJsonFilename PrivateValidatorData
+handledAsJsonFilename ValidatorPrivKey

beacon_chain/trusted_state_snapshots.nim

@@ -1,5 +1,5 @@
 import
-  chronos,
+  ospaths, chronos, json_serialization, strformat,
   spec/[datatypes, crypto, digest, beaconstate], beacon_chain_db, conf
 
 const
@@ -30,13 +30,18 @@ proc obtainTrustedStateSnapshot*(db: BeaconChainDB): Future[BeaconState] {.async
   assert(false, "Not implemented")
 
 proc createStateSnapshot*(
-    startup: ChainStartupData, genesisOffset: int, outFile: string) =
+    validatorDir: string, numValidators, firstValidator, genesisOffset: int,
+    outFile: string) =
+
+  var deposits: seq[Deposit]
+  for i in firstValidator..<numValidators:
+    deposits.add Json.loadFile(validatorDir / &"v{i:07}.deposit.json", Deposit)
+
   let initialState = get_genesis_beacon_state(
-    startup.validatorDeposits,
+    deposits,
     uint64(int(fastEpochTime() div 1000) + genesisOffset),
     Eth1Data(), {})
 
   var vr: Validator
   Json.saveFile(outFile, initialState, pretty = true)
   echo "Wrote ", outFile
-

beacon_chain/validator_keygen.nim

@@ -8,38 +8,21 @@ proc writeFile(filename: string, value: auto) =
   Json.saveFile(filename, value, pretty = true)
   echo "Wrote ", filename
 
-proc genSingleValidator(path: string): (ValidatorPubKey,
+cli do (validators: int = 100000,
-                                        ValidatorPrivKey) =
+        outputDir: string = "validators"):
-  var v: PrivateValidatorData
-  v.privKey = newPrivKey()
 
-  writeFile(path, v)
+  for i in 0 ..< validators:
+    # there can apparently be tops 4M validators so we use 7 digits..
+    let
+      depositFn = outputDir / &"v{i:07}.deposit.json"
+      privKeyFn = outputDir / &"v{i:07}.privkey.json"
 
-  assert v.privKey != ValidatorPrivKey(), "Private key shouldn't be zero"
+    if existsFile(depositFn) and existsFile(privKeyFn):
-  return (v.privKey.pubKey(), v.privKey)
+      continue
 
-# TODO: Make these more comprehensive and find them a new home
+    let
-type
+      privKey = makeFakeValidatorPrivKey(i)
-  Ether* = distinct int64
+      pubKey = privKey.pubKey()
-  GWei* = distinct int64
-
-template eth*(x: SomeInteger): Ether = Ether(x)
-template gwei*(x: Ether): Gwei = Gwei(int(x) * 1000000000)
-
-cli do (validators: int,
-        outputDir: string,
-        startupDelay = 0):
-
-  if validators < 64:
-    echo "The number of validators must be higher than ", SLOTS_PER_EPOCH, " (SLOTS_PER_EPOCH)"
-    echo "There must be at least one validator assigned per slot."
-    quit 1
-
-  var startupData: ChainStartupData
-
-  for i in 1 .. validators:
-    let (pubKey, privKey) =
-      genSingleValidator(outputDir / &"validator-{i:02}.json")
 
     let
       withdrawalCredentials = makeFakeHash(i)
@@ -53,14 +36,17 @@ cli do (validators: int,
         0 # TODO - domain
         )
 
-    startupData.validatorDeposits.add Deposit(
+    let
-      deposit_data: DepositData(
+      deposit = Deposit(
-        amount: MAX_DEPOSIT_AMOUNT,
+        deposit_data: DepositData(
-        timestamp: now(),
+          amount: MAX_DEPOSIT_AMOUNT,
-        deposit_input: DepositInput(
+          timestamp: now(),
-          pubkey: pubKey,
+          deposit_input: DepositInput(
-          proof_of_possession: proofOfPossession,
+            pubkey: pubKey,
-          withdrawal_credentials: withdrawalCredentials)))
+            proof_of_possession: proofOfPossession,
+            withdrawal_credentials: withdrawalCredentials)))
 
-  writeFile(outputDir / "startup.json", startupData)
+    writeFile(privKeyFn, privKey)
+    writeFile(depositFn, deposit)
 
+  echo "Keys generated by this tool are only for testing!"

research/state_sim.nim

@@ -106,11 +106,11 @@ cli do(slots = 1945,
           for v in scas.committee:
             if (rand(r, high(int)).float * attesterRatio).int <= high(int):
               if first:
-                attestation = makeAttestation(state, latest_block_root, v)
+                attestation = makeAttestation(state, latest_block_root, v, flags)
                 first = false
               else:
                 attestation.combine(
-                  makeAttestation(state, latest_block_root, v), flags)
+                  makeAttestation(state, latest_block_root, v, flags), flags)
 
         if not first:
           # add the attestation if any of the validators attested, as given
@@ -138,6 +138,10 @@ cli do(slots = 1945,
   echo "All time are ms"
   echo &"{\"Average\" :>12}, {\"StdDev\" :>12}, {\"Min\" :>12}, " &
     &"{\"Max\" :>12}, {\"Samples\" :>12}, {\"Test\" :>12}"
+
+  if not validate:
+    echo "Validation is turned off meaning that no BLS operations are performed"
+
   for t in Timers:
     echo fmtTime(timers[t].mean), fmtTime(timers[t].standardDeviationS),
       fmtTime(timers[t].min), fmtTime(timers[t].max), &"{timers[t].n :>12}, ",

tests/simulation/.gitignore

@@ -1,3 +1,3 @@
 data/
-startup/
+validators/
 

tests/simulation/run_node.sh

@@ -12,18 +12,22 @@ fi
 
 DATA_DIR=$SIMULATION_DIR/node-${1}
 
+V_PREFIX="$VALIDATORS_DIR/v$(printf '%06d' ${1})"
+PORT=$(printf '5%04d' ${1})
+
 $BEACON_NODE_BIN \
   --dataDir:$DATA_DIR \
-  --validator:$STARTUP_DIR/validator-${1}1.json \
+  --validator:${V_PREFIX}0.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}2.json \
+  --validator:${V_PREFIX}1.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}3.json \
+  --validator:${V_PREFIX}2.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}4.json \
+  --validator:${V_PREFIX}3.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}5.json \
+  --validator:${V_PREFIX}4.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}6.json \
+  --validator:${V_PREFIX}5.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}7.json \
+  --validator:${V_PREFIX}6.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}8.json \
+  --validator:${V_PREFIX}7.privkey.json \
-  --validator:$STARTUP_DIR/validator-${1}9.json \
+  --validator:${V_PREFIX}8.privkey.json \
-  --tcpPort:5000${1} \
+  --validator:${V_PREFIX}9.privkey.json \
-  --udpPort:5000${1} \
+  --tcpPort:$PORT \
+  --udpPort:$PORT \
   --stateSnapshot:$SNAPSHOT_FILE \
   $BOOTSTRAP_NODES_FLAG

tests/simulation/start.sh

@@ -10,11 +10,12 @@ trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 
 # Set a default value for the env vars usually supplied by nimbus Makefile
 
-NUMBER_OF_VALIDATORS=99
+NUM_VALIDATORS=${1:-100}
+NUM_NODES=${2:-9}
 
 cd $SIM_ROOT
 mkdir -p "$SIMULATION_DIR"
-mkdir -p "$STARTUP_DIR"
+mkdir -p "$VALIDATORS_DIR"
 
 cd $GIT_ROOT
 mkdir -p $BUILD_OUTPUTS_DIR
@@ -24,12 +25,15 @@ DEFS="-d:SHARD_COUNT=${SHARD_COUNT:-4} "      # Spec default: 1024
 DEFS+="-d:SLOTS_PER_EPOCH=${SLOTS_PER_EPOCH:-8} "   # Spec default: 64
 DEFS+="-d:SECONDS_PER_SLOT=${SECONDS_PER_SLOT:-6} " # Spec default: 6
 
-if [ ! -f $STARTUP_FILE ]; then
+LAST_VALIDATOR_NUM=$(( $NUM_VALIDATORS - 1 ))
+LAST_VALIDATOR="$VALIDATORS_DIR/v$(printf '%07d' $LAST_VALIDATOR_NUM).deposit.json"
+
+if [ ! -f $LAST_VALIDATOR ]; then
   if [[ -z "$SKIP_BUILDS" ]]; then
     nim c -o:"$VALIDATOR_KEYGEN_BIN" $DEFS -d:release beacon_chain/validator_keygen
   fi
 
-  $VALIDATOR_KEYGEN_BIN --validators=$NUMBER_OF_VALIDATORS --outputDir="$STARTUP_DIR"
+  $VALIDATOR_KEYGEN_BIN --validators=$NUM_VALIDATORS --outputDir="$VALIDATORS_DIR"
 fi
 
 if [[ -z "$SKIP_BUILDS" ]]; then
@@ -38,8 +42,10 @@ fi
 
 if [ ! -f $SNAPSHOT_FILE ]; then
   $BEACON_NODE_BIN createChain \
-    --chainStartupData:$STARTUP_FILE \
+    --validatorsDir:$VALIDATORS_DIR \
-    --out:$SNAPSHOT_FILE --genesisOffset=5 # Delay in seconds
+    --out:$SNAPSHOT_FILE \
+    --numValidators=$NUM_VALIDATORS \
+    --genesisOffset=5 # Delay in seconds
 fi
 
 # Delete any leftover address files from a previous session
@@ -53,7 +59,9 @@ USE_MULTITAIL="${USE_MULTITAIL:-no}" # make it an opt-in
 type "$MULTITAIL" &>/dev/null || USE_MULTITAIL="no"
 COMMANDS=()
 
-for i in $(seq 0 8); do
+LAST_NODE=$(( $NUM_NODES - 1 ))
+
+for i in $(seq 0 $LAST_NODE); do
   BOOTSTRAP_NODES_FLAG="--bootstrapNodesFile:$MASTER_NODE_ADDRESS_FILE"
 
   if [[ "$i" == "0" ]]; then

tests/simulation/vars.sh

@@ -14,8 +14,7 @@ GIT_ROOT="$($PWD_CMD)"
 : ${BUILD_OUTPUTS_DIR:="$GIT_ROOT/build"}
 
 SIMULATION_DIR="$SIM_ROOT/data"
-STARTUP_DIR="$SIM_ROOT/startup"
+VALIDATORS_DIR="$SIM_ROOT/validators"
-STARTUP_FILE="$STARTUP_DIR/startup.json"
 SNAPSHOT_FILE="$SIMULATION_DIR/state_snapshot.json"
 BEACON_NODE_BIN=$BUILD_OUTPUTS_DIR/beacon_node
 VALIDATOR_KEYGEN_BIN=$BUILD_OUTPUTS_DIR/validator_keygen

tests/testutil.nim

@@ -11,7 +11,7 @@ import
   ../beacon_chain/[beacon_chain_db, extras, ssz, state_transition, validator_pool],
   ../beacon_chain/spec/[beaconstate, crypto, datatypes, digest, helpers, validator]
 
-func makeValidatorPrivKey(i: int): ValidatorPrivKey =
+func makeFakeValidatorPrivKey*(i: int): ValidatorPrivKey =
   var i = i + 1 # 0 does not work, as private key...
   copyMem(result.x[0].addr, i.addr, min(sizeof(result.x), sizeof(i)))
 
@@ -24,14 +24,14 @@ func hackPrivKey(v: Validator): ValidatorPrivKey =
   copyMem(
     i.addr, v.withdrawal_credentials.data[0].unsafeAddr,
     min(sizeof(v.withdrawal_credentials.data), sizeof(i)))
-  makeValidatorPrivKey(i)
+  makeFakeValidatorPrivKey(i)
 
 func makeDeposit(i: int, flags: UpdateFlags): Deposit =
   ## Ugly hack for now: we stick the private key in withdrawal_credentials
   ## which means we can repro private key and randao reveal from this data,
   ## for testing :)
   let
-    privkey = makeValidatorPrivKey(i)
+    privkey = makeFakeValidatorPrivKey(i)
     pubkey = privkey.pubKey()
     withdrawal_credentials = makeFakeHash(i)