Added bin_utils

iceage.py

@@ -1,12 +1,12 @@
 import random
 import datetime
 
-diffs = [2285.34 * 10**12]
+diffs = [2801.27 * 10**12]
-hashpower = diffs[0] / 23.94
+hashpower = diffs[0] / 30.19
-times = [1504880987]
+times = [1506842691]
 
 
-for i in range(4251936, 6010000):
+for i in range(4326925, 6010000):
     blocktime = random.expovariate(hashpower / diffs[-1])
     adjfac = max(1 - int(blocktime / 10), -99) / 2048.
     newdiff = diffs[-1] * (1 + adjfac)

papers/casper-economics/casper_economics_basic.tex

@@ -63,7 +63,7 @@
 \maketitle
 
 \begin{abstract}
-We give an introduction to the incentives in the Casper the Friendly Finality Gadget protocol, and show how the protocol behaves under individual choice analysis, collective choice analysis and griefing factor analysis. We show that (i) the protocol is a Nash equilibrium assuming any individual validator's deposit makes up less than $\frac{1}{3}$ of the total, (ii) collectively, the validators lose from causing protocol faults, and there is a minimum ratio between the losses incurred by the validators and the seriousness of the fault, and (iii) the griefing factor can be bounded above by $1$, though we will prefer an alternative model that bounds the griefing factor at $2$ in exchange for other benefits. We also describe tradeoffs between protocol fairness and incentivization and fallbacks to extra-protocol resolution mechanisms such as market-driven chain splits.
+We give an introduction to the incentives in the Casper the Friendly Finality Gadget protocol, and show how the protocol behaves under individual choice analysis, collective choice analysis and griefing factor analysis. We show that (i) the protocol is a Nash equilibrium assuming any individual validator's deposit makes up less than $\frac{1}{3}$ of the total, (ii) collectively, the validators lose from causing protocol faults, and there is a minimum ratio between the losses incurred by the validators and the seriousness of the fault, and (iii) even for majority attackers, censorship (or ``griefing'') of minorities can be made expensive, although different mechanisms will be used to deal with small-scale attacks and large-scale attacks.
 
 We assume the "Casper the Friendly Finality Gadget" paper as a dependency.
 \end{abstract}
@@ -101,11 +101,13 @@ $$ $$
 
 The blockchain state maintains the \textit{current validator set} $V_c: v \rightarrow R^+$, a mapping of validators to their deposit sizes (non-negative real numbers) and the \textit{previous validator set} $V_p: v \rightarrow R^+$. The \textit{total current deposit size} is equal to $\sum_{v \in V} V_c[v]$, the sum of all deposits in the current validator set, and the \textit{total previous deposit size} is likewise equal to the $\sum_{v \in V} V_p[v]$. Validators can deposit $n$ coins to join both validator sets with deposit size $n$, and a validator with deposit size $n'$ can withdraw $n'$ coins with a delay. For any deposit or withdraw action to fully take effect, three checkpoints need to be finalized in a chain after the withdraw is included in that chain (validators get inducted to and ejected from the current validator set first, so after two finalized hashes, a validator will be in one validator set but not the other).
 
-An \textit{epoch} is a range of 100 blocks (e.g. blocks 600...699 are epoch 6), and a \textit{checkpoint} as the hash of a block right before the start of an epoch. The \textit{epoch of a checkpoint} is the epoch \textit{after} the checkpoint, e.g. the epoch of a checkpoint which is the hash of some block 599 is 6.
+A checkpoint is either the genesis, or a block with block number $i * 100 + 99$ for some integer $i \ge 0$. An \emph{epoch} is defined as the contiguous sequence of blocks between two checkpoints, including the later checkpoint but not the earlier one.  The \textit{epoch of a block} is the index of the epoch containing that hash, e.g., the epoch of block 599 is 5.
 
 We will use ``$p$ of validators'' for any fraction $p$ (eg. $\frac{2}{3}$) as shorthand for ``some set of validators $V_s$ such that $\sum_{v \in V_s} V_c[v] \ge \sum_{v \in V_c} V_c[v] * p$ and $\sum_{v \in V_s} V_p[v] \ge \sum_{v \in V_p} V_p[v] * p$'' - that is, such a set must make up \textit{both} $p$ of the current validator set \textit{and} $p$ of the previous validator set. The ``portion of validators that did X'' refers to the largest value $0 \le p \le 1$ such that $p$ of validators (using the definition above) did X.
 
-Every checkpoint hash $\hash$ has one of three possible states: \emph{fresh}, \emph{justified}, and \emph{finalized}.  Every hash starts as \emph{fresh}. A hash $\hash$ converts from fresh to \emph{justified} if $\frac{2}{3}$ of validators send prepares for $\hash$ with the same $(\epoch, \hashsource, \epochsource)$ triplet. An ``ideal execution'' of the protocol is one where, at the start of every epoch, every validator prepares and commits the same checkpoint for that epoch, specifying the same $\epochsource$ and $\hashsource$; thus, in every epoch, that checkpoint gets finalized. We wish to incentivize this ideal execution.
+A hash \hash is justified if, for some \hashsource (which must be an ancestor of \hash), there exist $\frac{2}{3}$ prepares of the form $\langle \msgPREPARE, \hash, \epoch(\hash), \hashsource, \epoch(\hashsource), \signature \rangle$, and \hashsource is itself justified. A hash \hash is finalized if it is justified and there exist $\frac{2}{3}$ commits of the form $\langle \msgCOMMIT, \hash, \epoch(\hash), \signature \rangle$. The genesis is considered both justified and finalized, and serves as the base case for the recursive definition.
+
+An ``ideal execution'' of the protocol is one where, at the start of every epoch, every validator prepares and commits the same checkpoint for that epoch, specifying the same $\epochsource$ and $\hashsource$; thus, in every epoch, that checkpoint gets finalized. We wish to incentivize this ideal execution.
 
 Possible deviations from this ideal execution that we want to minimize or avoid include:
 
@@ -158,11 +160,13 @@ The list above is deliberately organized symmetrically, to illustrate a fundamen
 
 What this means is that, in a liveness fault, we cannot unambiguously determine who was at fault, and this creates a fundamental tension between \textit{disincentivizing harm} and \textit{fairness} - between sufficiently penalizing validators who are malicious and not excessively penalizing validators who are not at fault. A protocol which absolutely ensures that innocent validators will not lose money must thus rely only on rewards, not on penalties, for discouraging non-uniquely-attributable faults, and so will only have a cryptoeconomic security margin equal to the size of the rewards that it issues. A protocol that penalizes suspected validators to the maximum will be one where innocent validators will not feel comfortable participating, which itself reduces security.
 
-A third ``way out'' is punting to off-chain governance. If a fault could have been caused by either A or B, then split the chain in half, on one branch penalize A, on the other branch penalize B, and let the market sort it out. We can theorize that the market will prefer branches where malfeasant validators control a smaller portion of the validator set, and so on the chain that ``wins'' the validators that the market subjectively deems to have been responsible for the fault will lose money and the innocent valdidators will not.
+We introduce a third axis that we can trade off: delegating to off-chain governance in the form of an inter-fork market. If a fault could have been caused by either A or B, then split the chain into two. On one branch penalize A, on the other branch penalize B, and let the market sort it out. We can theorize that the market will prefer branches where malfeasant validators control a smaller portion of the validator set, and so on the chain that ``wins'' the validators that the market subjectively deems to have been responsible for the fault will lose money and the innocent valdidators will not. Hence, we treat the market as a kind of magic oracle.
 
 [diagram]
 
-However, there must be some cost to triggering a ``governance event''; otherwise, attackers could trigger these events as a deliberate strategy in order to breed continual chaos among the users of a blockchain. The social value of blockchains largely comes from the fact that their progression is mostly automated, and so the more we can reduce the need for users to appeal to the social layer the better.
+However, there must be some high cost to invoking this oracle; otherwise, attackers could trigger these events as a deliberate strategy in order to breed continual chaos among the users of a blockchain. The social value of blockchains largely comes from the fact that the protocol execution is mostly automated and independent of human judgement, and so the more we can reduce the need for users to appeal to the social layer the better.
+
+Note that this use of the ``market oracle'' is NOT a special feature of proof of stake that does not exist in proof of work. Proof of work blockchain protocols generally have no formal treatment at all of how to recover from 51\% attacks, beyond informal references to the possibility of changing the proof of work [cite]. This is also an appeal to the ``market oracle'', albeit one that is vastly inferior to what we introduce here because (i) changing the proof of work can only be done once, as after the first time it is done no one will have specialized hardware, and if the attacker also corners the market on general-purpose hardware then there really is no way out, and (ii) in a proof of work change, both honest and malicious miners suffer a total capital loss, whereas in proof of stake we can largely ensure that only malicious validators lose out.
 
 \section{Rewards and Penalties, Not Penalizing Majority Censorship}
 

trie_research/bin_utils.py

@@ -0,0 +1,56 @@
+from ethereum.utils import safe_ord as ord
+
+# 0100000101010111010000110100100101001001 -> ASCII
+def decode_bin(x):
+    o = bytearray(len(x) // 8)
+    for i in range(0, len(x), 8):
+        v = 0
+        for c in x[i:i+8]:
+            v = v * 2 + c
+        o[i//8] = v
+    return bytes(o)
+
+
+# ASCII -> 0100000101010111010000110100100101001001
+def encode_bin(x):
+    o = b''
+    for c in x:
+        c = ord(c)
+        p = bytearray(8)
+        for i in range(8):
+            p[7-i] = c % 2
+            c //= 2
+        o += p
+    return o
+
+two_bits = [bytes([0,0]), bytes([0,1]),
+            bytes([1,0]), bytes([1,1])]
+prefix00 = bytes([0,0])
+prefix100000 = bytes([1,0,0,0,0,0])
+
+
+# Encodes a sequence of 0s and 1s into tightly packed bytes
+def encode_bin_path(b):
+    b2 = bytes((4 - len(b)) % 4) + b
+    prefix = two_bits[len(b) % 4]
+    if len(b2) % 8 == 4:
+        return decode_bin(prefix00 + prefix + b2)
+    else:
+        return decode_bin(prefix100000 + prefix + b2)
+
+
+# Decodes bytes into a sequence of 0s and 1s
+def decode_bin_path(p):
+    p = encode_bin(p)
+    if p[0] == 1:
+        p = p[4:]
+    assert p[0:2] == prefix00
+    L = two_bits.index(p[2:4])
+    return p[4+((4 - L) % 4):]
+
+def common_prefix_length(a, b):
+    o = 0
+    while o < len(a) and o < len(b) and a[o] == b[o]:
+        o += 1
+    return o
+