</a></div><!----></nav><ulclass="sidebar-links"><li><sectionclass="sidebar-group depth-0"><pclass="sidebar-heading open"><span>Eth</span><!----></p><ulclass="sidebar-links sidebar-group-items"><li><ahref="/lib/nim-eth/#rlp"class="sidebar-link">rlp</a><ulclass="sidebar-sub-headers"><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#introduction"class="sidebar-link">Introduction</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#reading-rlp-data"class="sidebar-link">Reading RLP data</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#streaming-api"class="sidebar-link">Streaming API</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#dom-api"class="sidebar-link">DOM API</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#creating-rlp-data"class="sidebar-link">Creating RLP data</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#object-serialization"class="sidebar-link">Object serialization</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#contributing-testing"class="sidebar-link">Contributing / Testing</a></li></ul></li><li><ahref="/lib/nim-eth/#p2p"class="sidebar-link">p2p</a><ulclass="sidebar-sub-headers"><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#introduction-2"class="sidebar-link">Introduction</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#connecting-to-the-ethereum-network"class="sidebar-link">Connecting to the Ethereum network</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#communicating-with-peers-using-rlpx"class="sidebar-link">Communicating with Peers using RLPx</a></li></ul></li><li><ahref="/lib/nim-eth/#keys"class="sidebar-link">keys</a><ulclass="sidebar-sub-headers"></ul></li><li><ahref="/lib/nim-eth/#keyfile"class="sidebar-link">keyfile</a><ulclass="sidebar-sub-headers"><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#introduction-3"class="sidebar-link">Introduction</a></li></ul></li><li><ahref="/lib/nim-eth/#trie"class="sidebar-link">trie</a><ulclass="sidebar-sub-headers"></ul></li><li><ahref="/lib/nim-eth/#nim-implementation-of-the-ethereum-trie-structure"class="sidebar-link">Nim Implementation of the Ethereum Trie structure</a><ulclass="sidebar-sub-headers"><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#hexary-trie"class="sidebar-link">Hexary Trie</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#binary-trie"class="sidebar-link">Binary Trie</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#examples"class="sidebar-link">Examples</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#the-truth-behind-a-lie"class="sidebar-link">The truth behind a lie</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#the-branch-utils"class="sidebar-link">The branch utils</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#remember-the-lie"class="sidebar-link">Remember the lie?</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#sparse-merkle-trie"class="sidebar-link">Sparse Merkle Trie</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#examples-2"class="sidebar-link">Examples</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#merkle-proofing"class="sidebar-link">Merkle Proofing</a></li></ul></li><li><ahref="/lib/nim-eth/#bloom-an-ethereum-bloom-filter"class="sidebar-link">bloom: an Ethereum Bloom Filter</a><ulclass="sidebar-sub-headers"></ul></li><li><ahref="/lib/nim-eth/#introduction-4"class="sidebar-link">Introduction</a><ulclass="sidebar-sub-headers"></ul></li><li><ahref="/lib/nim-eth/#description"class="sidebar-link">Description</a><ulclass="sidebar-sub-headers"></ul></li><li><ahref="/lib/nim-eth/#usage"class="sidebar-link">Usage</a><ulclass="sidebar-sub-headers"></ul></li><li><ahref="/lib/nim-eth/#node-discovery-protocol-v5"class="sidebar-link">Node Discovery Protocol v5</a><ulclass="sidebar-sub-headers"><liclass="sidebar-sub-header"><ahref="/lib/nim-eth/#introduction-5"class="sidebar-link">Introduction</a></li><liclass="sidebar-sub-header"><ahref="/lib/nim-
in the Ethereum's <ahref="https://ethereum.github.io/yellowpaper/paper.pdf"target="_blank"rel="noopener noreferrer">Yellow Paper<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>
and <ahref="https://github.com/ethereum/wiki/wiki/RLP"target="_blank"rel="noopener noreferrer">Wiki<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>.</p><h3id="reading-rlp-data"><ahref="#reading-rlp-data"class="header-anchor">#</a> Reading RLP data</h3><p>The <code>Rlp</code> type provided by this library represents a cursor over an RLP-encoded
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br></div></div><h3id="streaming-api"><ahref="#streaming-api"class="header-anchor">#</a> Streaming API</h3><p>Once created, the <code>Rlp</code> object will offer procs such as <code>isList</code>, <code>isBlob</code>,
<code>getType</code>, <code>listLen</code>, <code>blobLen</code> to determine the type of the value under
the cursor. The contents of blobs can be extracted with procs such as
<code>toString</code>, <code>toBytes</code> and <code>toInt</code> without advancing the cursor.</p><p>Lists can be traversed with the standard <code>items</code> iterator, which will advance
the cursor to each sub-item position and yield the <code>Rlp</code> object at that point.
As an alternative, <code>listElem</code> can return a new <code>Rlp</code> object adjusted to a
particular sub-item position without advancing the original cursor.
Keep in mind that copying <code>Rlp</code> objects is cheap and you can create as many
cursors pointing to different positions in the RLP stream as necessary.</p><p><code>skipElem</code> will advance the cursor to the next position in the current list.
<code>hasData</code> will indicate that there are no more bytes in the stream that can
be consumed.</p><p>Another way to extract data from the stream is through the universal <code>read</code>
proc that accepts a type as a parameter. You can pass any supported type
such as <code>string</code>, <code>int</code>, <code>seq[T]</code>, etc, including composite user-defined
types (see <ahref="#object-serialization">Object Serialization</a>). The cursor
will be advanced just past the end of the consumed object.</p><p>The <code>toXX</code> and <code>read</code> family of procs may raise a <code>RlpTypeMismatch</code> in case
of type mismatch with the stream contents under the cursor. A corrupted
RLP stream or an attemp to read past the stream end will be signaled
with the <code>MalformedRlpError</code> exception. If the RLP stream includes data
that cannot be processed on the current platform (e.g. an integer value
that is too large), the library will raise an <code>UnsupportedRlpError</code> exception.</p><h3id="dom-api"><ahref="#dom-api"class="header-anchor">#</a> DOM API</h3><p>Calling <code>Rlp.toNodes</code> at any position within the stream will return a tree
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br></div></div><p>As a short-cut, you can also call <code>decode</code> directly on a byte sequence to
avoid creating a <code>Rlp</code> object when obtaining the nodes.
For debugging purposes, you can also create a human readable representation
of the Rlp nodes by calling the <code>inspect</code> proc:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">proc</span><spanclass="token function">inspect<spanclass="token operator">*</span></span><spanclass="token punctuation">(</span>self<spanclass="token operator">:</span> Rlp<spanclass="token punctuation">,</span> indent <spanclass="token operator">=</span><spanclass="token number">0</span><spanclass="token punctuation">)</span><spanclass="token operator">:</span> string
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br></div></div><h3id="creating-rlp-data"><ahref="#creating-rlp-data"class="header-anchor">#</a> Creating RLP data</h3><p>The <code>RlpWriter</code> type can be used to encode RLP data. Instances are created
with the <code>initRlpWriter</code> proc. This should be followed by one or more calls
to <code>append</code> which is overloaded to accept arbitrary values. Finally, you can
call <code>finish</code> to obtain the final <code>seq[byte]</code>.</p><p>If the end result should be a RLP list of particular length, you can replace
the initial call to <code>initRlpWriter</code> with <code>initRlpList(n)</code>. Calling <code>finish</code>
before writing the sufficient number of elements will then result in an assertion failure.</p><p>As an alternative short-cut, you can also call <code>encode</code> on an arbitrary value
(including sequences and user-defined types) to execute all of the steps at
once and directly obtain the final RLP bytes. <code>encodeList(varargs)</code> is another
short-cut for creating RLP lists.</p><h3id="object-serialization"><ahref="#object-serialization"class="header-anchor">#</a> Object serialization</h3><p>As previously explained, generic procs such as <code>read</code>, <code>append</code>, <code>encode</code> and
<code>decode</code> can be used with arbitrary used-defined object types. By default, the
library will serialize all of the fields of the object using the <code>fields</code>
iterator, but you can also include only a subset of the fields or modify the
order of serialization or by employing the <code>rlpIgnore</code> pragma or by using the
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br><spanclass="line-number">12</span><br><spanclass="line-number">13</span><br><spanclass="line-number">14</span><br><spanclass="line-number">15</span><br><spanclass="line-number">16</span><br><spanclass="line-number">17</span><br><spanclass="line-number">18</span><br><spanclass="line-number">19</span><br></div></div><p>By default, sub-fields within objects are wrapped in RLP lists. You can avoid this
behavior by adding the custom pragma <code>rlpInline</code> on a particular field. In rare
circumstances, you may need to serialize the same field type differently depending
on the enclosing object type. You can use the <code>rlpCustomSerialization</code> pragma to
achieve this.</p><h3id="contributing-testing"><ahref="#contributing-testing"class="header-anchor">#</a> Contributing / Testing</h3><p>To test the correctness of any modifications to the library, please execute
<code>nimble test_rlp</code> at the root of the repo.</p><h2id="p2p"><ahref="#p2p"class="header-anchor">#</a> p2p</h2><h3id="introduction-2"><ahref="#introduction-2"class="header-anchor">#</a> Introduction</h3><p>This library implements the DevP2P family of networking protocols used
in the Ethereum world.</p><h3id="connecting-to-the-ethereum-network"><ahref="#connecting-to-the-ethereum-network"class="header-anchor">#</a> Connecting to the Ethereum network</h3><p>A connection to the Ethereum network can be created by instantiating
A name used to identify the software package connecting
to the network (i.e. similar to the <code>User-Agent</code> string
in a browser).</p><p><code>addAllCapabilities</code>:
By default, the node will support all RPLx protocols imported in
your project. You can specify <code>false</code> if you prefer to create a
node with a more limited set of protocols. Use one or more calls
to <code>node.addCapability</code> to specify the desired set:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code>node<spanclass="token operator">.</span><spanclass="token function">addCapability</span><spanclass="token punctuation">(</span>eth<spanclass="token punctuation">)</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br></div></div><p>Each supplied protocol identifier is a name of a protocol introduced
by the <code>p2pProtocol</code> macro discussed later in this document.</p><p>Instantiating an <code>EthereumNode</code> does not immediately connect you to
the network. To start the connection process, call <code>node.connectToNetwork</code>:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">proc</span><spanclass="token function">connectToNetwork<spanclass="token operator">*</span></span><spanclass="token punctuation">(</span>node<spanclass="token operator">:</span><spanclass="token keyword">var</span> EthereumNode<spanclass="token punctuation">,</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br></div></div><p>The <code>EthereumNode</code> will automatically find and maintain a pool of peers
using the Ethereum node discovery protocol. You can access the pool as
<code>node.peers</code>.</p><h3id="communicating-with-peers-using-rlpx"><ahref="#communicating-with-peers-using-rlpx"class="header-anchor">#</a> Communicating with Peers using RLPx</h3><p><ahref="https://github.com/ethereum/devp2p/blob/master/rlpx.md"target="_blank"rel="noopener noreferrer">RLPx<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a> is the
high-level protocol for exchanging messages between peers in the Ethereum
network. Most of the client code of this library should not be concerned
with the implementation details of the underlying protocols and should use
the high-level APIs described in this section.</p><p>The RLPx protocols are defined as a collection of strongly-typed messages,
which are grouped into sub-protocols multiplexed over the same TCP connection.</p><p>This library represents each such message as a regular Nim function call
over the <code>Peer</code> object. Certain messages act only as notifications, while
others fit the request/response pattern.</p><p>To understand more about how messages are defined and used, let's look at
the definition of a RLPx protocol:</p><h4id="rlpx-sub-protocols"><ahref="#rlpx-sub-protocols"class="header-anchor">#</a> RLPx sub-protocols</h4><p>The sub-protocols are defined with the <code>p2pProtocol</code> macro. It will accept
a short identifier for the protocol and the current protocol version:</p><p>Here is how the <ahref="https://github.com/ethereum/devp2p/blob/master/rlpx.md#p2p-capability"target="_blank"rel="noopener noreferrer">DevP2P wire protocol<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a> might look like:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code>p2pProtocol <spanclass="token function">DevP2P</span><spanclass="token punctuation">(</span>version <spanclass="token operator">=</span><spanclass="token number">0</span><spanclass="token punctuation">,</span> rlpxName <spanclass="token operator">=</span><spanclass="token string">"p2p"</span><spanclass="token punctuation">)</span><spanclass="token operator">:</span>
echo <spanclass="token string">"received pong from "</span><spanclass="token punctuation">,</span> peer<spanclass="token operator">.</span>id
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br><spanclass="line-number">12</span><br><spanclass="line-number">13</span><br><spanclass="line-number">14</span><br><spanclass="line-number">15</span><br><spanclass="line-number">16</span><br></div></div><p>As seen in the example above, a protocol definition determines both the
available messages that can be sent to another peer (e.g. as in <code>peer.pong()</code>)
and the asynchronous code responsible for handling the incoming messages.</p><h4id="protocol-state"><ahref="#protocol-state"class="header-anchor">#</a> Protocol state</h4><p>The protocol implementations are expected to maintain a state and to act
like a state machine handling the incoming messages. You are allowed to
define an arbitrary state type that can be specified in the <code>peerState</code>
protocol option. Later, instances of the state object can be obtained
though the <code>state</code> pseudo-field of the <code>Peer</code> object:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">type</span> AbcPeerState <spanclass="token operator">=</span><spanclass="token keyword">object</span>
receivedMsgsCount<spanclass="token operator">:</span> int
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br></div></div><p>Besides the per-peer state demonstrated above, there is also support
for maintaining a network-wide state. It's enabled by specifying the
<code>networkState</code> option of the protocol and the state object can be obtained
through accessor of the same name.</p><p>The state objects are initialized to zero by default, but you can modify
this behaviour by overriding the following procs for your state types:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">proc</span><spanclass="token function">initProtocolState<spanclass="token operator">*</span></span><spanclass="token punctuation">(</span>state<spanclass="token operator">:</span> MyPeerState<spanclass="token punctuation">,</span> p<spanclass="token operator">:</span> Peer<spanclass="token punctuation">)</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br></div></div><p>Sometimes, you'll need to access the state of another protocol.
To do this, specify the protocol identifier to the <code>state</code> accessors:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code> echo <spanclass="token string">"ABC protocol messages: "</span><spanclass="token punctuation">,</span> peer<spanclass="token operator">.</span><spanclass="token function">state</span><spanclass="token punctuation">(</span>abc<spanclass="token punctuation">)</span><spanclass="token operator">.</span>receivedMsgCount
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br></div></div><p>While the state machine approach may be a particularly robust way of
implementing sub-protocols (it is more amenable to proving the correctness
of the implementation through formal verification methods), sometimes it may
be more convenient to use more imperative style of communication where the
code is able to wait for a particular response after sending a particular
request. The library provides two mechanisms for achieving this:</p><h4id="waiting-particular-messages-with-nextmsg"><ahref="#waiting-particular-messages-with-nextmsg"class="header-anchor">#</a> Waiting particular messages with <code>nextMsg</code></h4><p>The <code>nextMsg</code> helper proc can be used to pause the execution of an async
proc until a particular incoming message from a peer arrives:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">proc</span><spanclass="token function">helloExample</span><spanclass="token punctuation">(</span>peer<spanclass="token operator">:</span> Peer<spanclass="token punctuation">)</span><spanclass="token operator">=</span>
<spanclass="token operator">...</span>
<spanclass="token comment"># send a hello message</span>
echo response<spanclass="token operator">.</span>clientId <spanclass="token comment"># print the name of the Ethereum client</span>
<spanclass="token comment"># used by the other peer (Geth, Parity, Nimbus, etc)</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br></div></div><p>There are few things to note in the above example:</p><ol><li><p>The <code>p2pProtocol</code> definition created a pseudo-variable named after the
protocol holding various properties of the protocol.</p></li><li><p>Each message defined in the protocol received a corresponding type name,
matching the message name (e.g. <code>p2p.hello</code>). This type will have fields
matching the parameter names of the message. If the messages has <code>openarray</code>
params, these will be remapped to <code>seq</code> types.</p></li></ol><p>If the designated messages also has an attached handler, the future returned
by <code>nextMsg</code> will be resolved only after the handler has been fully executed
(so you can count on any side effects produced by the handler to have taken
place). If there are multiple outstanding calls to <code>nextMsg</code>, they will
complete together. Any other messages received in the meantime will still
be dispatched to their respective handlers.</p><p>Please also note that the <code>p2pProtocol</code> macro will make this <code>helloExample</code> proc
<code>async</code>. Practically see it as <code>proc helloExample(peer: Peer) {.async.}</code>, and
thus never use <code>waitFor</code>, but rather <code>await</code> inside this proc.</p><p>For implementing protocol handshakes with <code>nextMsg</code> there are specific helpers
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br></div></div><p>Two or more messages within the protocol may be grouped into a
<code>requestResponse</code> block. The last message in the group is assumed
to be the response while all other messages are considered requests.</p><p>When a request message is sent, the return type will be a <code>Future</code>
that will be completed once the response is received. Please note
that there is a mandatory timeout parameter, so the actual return
type is <code>Future[Option[MessageType]]</code>. The <code>timeout</code> parameter can
be specified for each individual call and the default value can be
overridden on the level of individual message, or the entire protocol:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code>p2pProtocol <spanclass="token function">abc</span><spanclass="token punctuation">(</span>version <spanclass="token operator">=</span><spanclass="token number">1</span><spanclass="token punctuation">,</span>
timeout <spanclass="token operator">=</span><spanclass="token number">5000</span><spanclass="token punctuation">)</span><spanclass="token operator">:</span><spanclass="token comment"># value in milliseconds</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br></div></div><p>By default, the library will take care of inserting a hidden <code>reqId</code>
parameter as used in the <ahref="https://github.com/zsfelfoldi/go-ethereum/wiki/Light-Ethereum-Subprotocol-%28LES%29"target="_blank"rel="noopener noreferrer">LES protocol<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>,
but you can disable this behavior by overriding the protocol setting
<code>useRequestIds</code>.</p><h4id="implementing-handshakes-and-reacting-to-other-events"><ahref="#implementing-handshakes-and-reacting-to-other-events"class="header-anchor">#</a> Implementing handshakes and reacting to other events</h4><p>Besides message definitions and implementations, a protocol specification may
also include handlers for certain important events such as newly connected
peers or misbehaving or disconnecting peers:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code>p2pProtocol <spanclass="token function">foo</span><spanclass="token punctuation">(</span>version <spanclass="token operator">=</span> fooVersion<spanclass="token punctuation">)</span><spanclass="token operator">:</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br><spanclass="line-number">12</span><br><spanclass="line-number">13</span><br><spanclass="line-number">14</span><br><spanclass="line-number">15</span><br><spanclass="line-number">16</span><br></div></div><p>For handshake messages, where the same type of message needs to be send to and
received from the peer, a <code>handshake</code> helper is introduced, as you can see in
the code example above.</p><p>Thanks to the <code>handshake</code> helper the <code>status</code> message will both be send, and be
awaited for receival from the peer, with the defined timeout. In case no <code>status</code>
message is received within the defined timeout, an error will be raised which
will result in a disconnect from the peer.</p><p><strong>Note:</strong> Be aware that if currently one of the subprotocol <code>onPeerConnected</code>
calls fails, the client will be disconnected as <code>UselessPeer</code> but no
<code>onPeerDisconnect</code> calls are run.</p><h4id="checking-the-other-peer-s-supported-sub-protocols"><ahref="#checking-the-other-peer-s-supported-sub-protocols"class="header-anchor">#</a> Checking the other peer's supported sub-protocols</h4><p>Upon establishing a connection, RLPx will automatically negotiate the list of
mutually supported protocols by the peers. To check whether a particular peer
supports a particular sub-protocol, use the following code:</p><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">if</span> peer<spanclass="token operator">.</span><spanclass="token function">supports</span><spanclass="token punctuation">(</span>les<spanclass="token punctuation">)</span><spanclass="token operator">:</span><spanclass="token comment"># `les` is the identifier of the light clients sub-protocol</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br></div></div><h2id="keys"><ahref="#keys"class="header-anchor">#</a> keys</h2><p>This library is a Nim re-implementation of <ahref="https://github.com/ethereum/eth-keys"target="_blank"rel="noopener noreferrer">eth-keys<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>: the common API for working with Ethereum's public and private keys, signatures, and addresses.</p><p>By default, Nim eth-keys uses Bitcoin's <ahref="https://github.com/bitcoin-core/secp256k1"target="_blank"rel="noopener noreferrer">libsecp256k1<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a> as a backend. Make sure libsecp256k1 is available on your system.</p><p>An experimental pure Nim backend (Warning ⚠: do not use in production) is available with the compilation switch <code>-d:backend_native</code></p><h2id="keyfile"><ahref="#keyfile"class="header-anchor">#</a> keyfile</h2><h3id="introduction-3"><ahref="#introduction-3"class="header-anchor">#</a> Introduction</h3><p>This library is a Nim reimplementation of <ahref="https://github.com/ethereum/eth-keyfile"target="_blank"rel="noopener noreferrer">ethereum/eth-keyfile<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>, which is used to create and load Ethereum <code>keyfile</code> format and the tools for handling the format and for storing private keys. Currently, the library supports only the PBKDF2 method and does not support the Scrypt method.</p><h2id="trie"><ahref="#trie"class="header-anchor">#</a> trie</h2><h2id="nim-implementation-of-the-ethereum-trie-structure"><ahref="#nim-implementation-of-the-ethereum-trie-structure"class="header-anchor">#</a> Nim Implementation of the Ethereum Trie structure</h2><h3id="hexary-trie"><ahref="#hexary-trie"class="header-anchor">#</a> Hexary Trie</h3><h3id="binary-trie"><ahref="#binary-trie"class="header-anchor">#</a> Binary Trie</h3><p>Binary-trie is a dictionary-like data structure to store key-value pair.
Much like it's sibling Hexary-trie, the key-value pair will be stored into key-value flat-db.
The primary difference with Hexary-trie is, each node of Binary-trie only consist of one or two child,
while Hexary-trie node can contains up to 16 or 17 child-nodes.</p><p>Unlike Hexary-trie, Binary-trie store it's data into flat-db without using rlp encoding.
Binary-trie store its value using simple <strong>Node-Types</strong> encoding.
The encoded-node will be hashed by keccak_256 and the hash value will be the key to flat-db.
Each entry in the flat-db will looks like:</p><table><thead><tr><th>key</th><th>value</th></tr></thead><tbody><tr><td>32-bytes-keccak-hash</td><td>encoded-node(KV or BRANCH or LEAF encoded)</td></tr></tbody></table><h4id="node-types"><ahref="#node-types"class="header-anchor">#</a> Node-Types</h4><ul><li>KV = [0, encoded-key-path, 32 bytes hash of child]</li><li>BRANCH = [1, 32 bytes hash of left child, 32 bytes hash of right child]</li><li>LEAF = [2, value]</li></ul><p>The KV node can have BRANCH node or LEAF node as it's child, but cannot a KV node.
The internal algorithm will merge a KV(parent)->KV(child) into one KV node.
Every KV node contains encoded keypath to reduce the number of blank nodes.</p><p>The BRANCH node can have KV, BRANCH, or LEAF node as it's children.</p><p>The LEAF node is the terminal node, it contains the value of a key.</p><h4id="encoded-key-path"><ahref="#encoded-key-path"class="header-anchor">#</a> encoded-key-path</h4><p>While Hexary-trie encode the path using Hex-Prefix encoding, Binary-trie
encode the path using binary encoding, the scheme looks like this table below.</p><divclass="language-text line-numbers-mode"><preclass="language-text"><code> |--------- odd --------|
00mm yyyy xxxx xxxx xxxx xxxx
|------ even -----|
1000 00mm yyyy xxxx xxxx xxxx
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br></div></div><table><thead><tr><th>symbol</th><th>explanation</th></tr></thead><tbody><tr><td>xxxx</td><td>nibble of binary keypath in bits, 0 = left, 1 = right</td></tr><tr><td>yyyy</td><td>nibble contains 0-3 bits padding + binary keypath</td></tr><tr><td>mm</td><td>number of binary keypath bits modulo 4 (0-3)</td></tr><tr><td>00</td><td>zero zero prefix</td></tr><tr><td>1000</td><td>even numbered nibbles prefix</td></tr></tbody></table><p>if there is no padding, then yyyy bit sequence is absent, mm also zero.
yyyy = mm bits + padding bits must be 4 bits length.</p><h4id="the-api"><ahref="#the-api"class="header-anchor">#</a> The API</h4><p>The primary API for Binary-trie is <code>set</code> and <code>get</code>.</p><ul><li>set(key, value) --- <em>store a value associated with a key</em></li><li>get(key): value --- <em>get a value using a key</em></li></ul><p>Both <code>key</code> and <code>value</code> are of <code>seq[byte]</code> type. And they cannot have zero length.</p><p>Getting a non-existent key will return zero length seq[byte].</p><p>Binary-trie also provide dictionary syntax API for <code>set</code> and <code>get</code>.</p><ul><li>trie[key] = value -- same as <code>set</code></li><li>value = trie[key] -- same as <code>get</code></li><li>contains(key) a.k.a. <code>in</code> operator</li></ul><p>Additional APIs are:</p><ul><li>exists(key) -- returns <code>bool</code>, to check key-value existence -- same as contains</li><li>delete(key) -- remove a key-value from the trie</li><li>deleteSubtrie(key) -- remove a key-value from the trie plus all of it's subtrie
that starts with the same key prefix</li><li>rootNode() -- get root node</li><li>rootNode(node) -- replace the root node</li><li>getRootHash(): <code>KeccakHash</code> with <code>seq[byte]</code> type</li><li>getDB(): <code>DB</code> -- get flat-db pointer</li></ul><p>Constructor API:</p><ul><li>initBinaryTrie(DB, rootHash[optional]) -- rootHash has <code>seq[byte]</code> or KeccakHash type</li><li>init(BinaryTrie, DB, rootHash[optional])</li></ul><p>Normally you would not set the rootHash when constructing an empty Binary-trie.
Setting the rootHash occured in a scenario where you have a populated DB
with existing trie structure and you know the rootHash,
and then you want to continue/resume the trie operations.</p><h3id="examples"><ahref="#examples"class="header-anchor">#</a> Examples</h3><divclass="language-Nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">import</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br><spanclass="line-number">12</span><br><spanclass="line-number">13</span><br><spanclass="line-number">14</span><br><spanclass="line-number">15</span><br><spanclass="line-number">16</span><br><spanclass="line-number">17</span><br><spanclass="line-number">18</span><br></div></div><p>Remember, <code>set</code> and <code>get</code> are trie operations. A single <code>set</code> operation may invoke
more than one store/lookup operation into the underlying DB. The same is also happened to <code>get</code> operation,
it could do more than one flat-db lookup before it return the requested value.</p><h3id="the-truth-behind-a-lie"><ahref="#the-truth-behind-a-lie"class="header-anchor">#</a> The truth behind a lie</h3><p>What kind of lie? actually, <code>delete</code> and <code>deleteSubtrie</code> doesn't remove the
'deleted' node from the underlying DB. It only make the node inaccessible
from the user of the trie. The same also happened if you update the value of a key,
the old value node is not removed from the underlying DB.
A more subtle lie also happened when you add new entrie into the trie using <code>set</code> operation.
The previous hash of affected branch become obsolete and replaced by new hash,
the old hash become inaccessible to the user.
You may think that is a waste of storage space.
Luckily, we also provide some utilities to deal with this situation, the branch utils.</p><h3id="the-branch-utils"><ahref="#the-branch-utils"class="header-anchor">#</a> The branch utils</h3><p>The branch utils consist of these API:</p><ul><li>checkIfBranchExist(DB; rootHash; keyPrefix): bool</li><li>getBranch(DB; rootHash; key): branch</li><li>isValidBranch(branch, rootHash, key, value): bool</li><li>getWitness(DB; nodeHash; key): branch</li><li>getTrieNodes(DB; nodeHash): branch</li></ul><p><code>keyPrefix</code>, <code>key</code>, and <code>value</code> are bytes container with length greater than zero.
They can be openArray[byte].</p><p><code>rootHash</code> and <code>nodeHash</code> also bytes container,
but they have constraint: must be 32 bytes in length, and it must be a keccak_256 hash value.</p><p><code>branch</code> is a list of nodes, or in this case a <code>seq[seq[byte]]</code>.
A list? yes, the structure is stored along with the encoded node.
Therefore a list is enough to reconstruct the entire trie/branch.</p><divclass="language-Nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">import</span>
x <spanclass="token operator">=</span><spanclass="token function">getBranch</span><spanclass="token punctuation">(</span>db<spanclass="token punctuation">,</span> trie<spanclass="token operator">.</span><spanclass="token function">getRootHash</span><spanclass="token punctuation">(</span><spanclass="token punctuation">)</span><spanclass="token punctuation">,</span><spanclass="token string">"key5"</span><spanclass="token punctuation">)</span><spanclass="token comment"># there is still branch for non-exist key</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br><spanclass="line-number">12</span><br><spanclass="line-number">13</span><br><spanclass="line-number">14</span><br><spanclass="line-number">15</span><br><spanclass="line-number">16</span><br><spanclass="line-number">17</span><br><spanclass="line-number">18</span><br><spanclass="line-number">19</span><br><spanclass="line-number">20</span><br><spanclass="line-number">21</span><br><spanclass="line-number">22</span><br><spanclass="line-number">23</span><br><spanclass="line-number">24</span><br><spanclass="line-number">25</span><br><spanclass="line-number">26</span><br><spanclass="line-number">27</span><br><spanclass="line-number">28</span><br><spanclass="line-number">29</span><br><spanclass="line-number">30</span><br><spanclass="line-number">31</span><br><spanclass="line-number">32</span><br><spanclass="line-number">33</span><br><spanclass="line-number">34</span><br><spanclass="line-number">35</span><br><spanclass="line-number">36</span><br></div></div><h3id="remember-the-lie"><ahref="#remember-the-lie"class="header-anchor">#</a> Remember the lie?</h3><p>Because trie <code>delete</code>, <code>deleteSubtrie</code> and <code>set</code> operation create inaccessible nodes in the underlying DB,
we need to remove them if necessary. We already see that <code>wholeTrie = getWitness(db, trie.getRootHash(), "")</code>
will return the whole trie, a list of accessible nodes.
Then we can write the clean tree into a new DB instance to replace the old one.</p><h3id="sparse-merkle-trie"><ahref="#sparse-merkle-trie"class="header-anchor">#</a> Sparse Merkle Trie</h3><p>Sparse Merkle Trie(SMT) is a variant of Binary Trie which uses binary encoding to
represent path during trie travelsal. When Binary Trie uses three types of node,
SMT only use one type of node without any additional special encoding to store it's key-path.</p><p>Actually, it doesn't even store it's key-path anywhere like Binary Trie,
the key-path is stored implicitly in the trie structure during key-value insertion.</p><p>Because the key-path is not encoded in any special ways, the bits can be extracted directly from
the key without any conversion.</p><p>However, the key restricted to a fixed length because the algorithm demand a fixed height trie
to works properly. In this case, the trie height is limited to 160 level,
or the key is of fixed length 20 bytes (8 bits x 20 = 160).</p><p>To be able to use variable length key, the algorithm can be adapted slightly using hashed key before
constructing the binary key-path. For example, if using keccak256 as the hashing function,
then the height of the tree will be 256, but the key itself can be any length.</p><h4id="the-api-2"><ahref="#the-api-2"class="header-anchor">#</a> The API</h4><p>The primary API for Binary-trie is <code>set</code> and <code>get</code>.</p><ul><li>set(key, value, rootHash[optional]) --- <em>store a value associated with a key</em></li><li>get(key, rootHash[optional]): value --- <em>get a value using a key</em></li></ul><p>Both <code>key</code> and <code>value</code> are of <code>BytesRange</code> type. And they cannot have zero length.
You can also use convenience API <code>get</code> and <code>set</code> which accepts
<code>Bytes</code> or <code>string</code> (a <code>string</code> is conceptually wrong in this context
and may costlier than a <code>BytesRange</code>, but it is good for testing purpose).</p><p>rootHash is an optional parameter. When used, <code>get</code> will get a key from specific root,
and <code>set</code> will also set a key at specific root.</p><p>Getting a non-existent key will return zero length BytesRange or a zeroBytesRange.</p><p>Sparse Merkle Trie also provide dictionary syntax API for <code>set</code> and <code>get</code>.</p><ul><li>trie[key] = value -- same as <code>set</code></li><li>value = trie[key] -- same as <code>get</code></li><li>contains(key) a.k.a. <code>in</code> operator</li></ul><p>Additional APIs are:</p><ul><li>exists(key) -- returns <code>bool</code>, to check key-value existence -- same as contains</li><li>delete(key) -- remove a key-value from the trie</li><li>getRootHash(): <code>KeccakHash</code> with <code>BytesRange</code> type</li><li>getDB(): <code>DB</code> -- get flat-db pointer</li><li>prove(key, rootHash[optional]): proof -- useful for merkling</li></ul><p>Constructor API:</p><ul><li>initSparseBinaryTrie(DB, rootHash[optional])</li><li>init(SparseBinaryTrie, DB, rootHash[optional])</li></ul><p>Normally you would not set the rootHash when constructing an empty Sparse Merkle Trie.
Setting the rootHash occured in a scenario where you have a populated DB
with existing trie structure and you know the rootHash,
and then you want to continue/resume the trie operations.</p><h3id="examples-2"><ahref="#examples-2"class="header-anchor">#</a> Examples</h3><divclass="language-Nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">import</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br><spanclass="line-number">12</span><br><spanclass="line-number">13</span><br><spanclass="line-number">14</span><br><spanclass="line-number">15</span><br><spanclass="line-number">16</span><br><spanclass="line-number">17</span><br><spanclass="line-number">18</span><br><spanclass="line-number">19</span><br><spanclass="line-number">20</span><br><spanclass="line-number">21</span><br></div></div><p>Remember, <code>set</code> and <code>get</code> are trie operations. A single <code>set</code> operation may invoke
more than one store/lookup operation into the underlying DB. The same is also happened to <code>get</code> operation,
it could do more than one flat-db lookup before it return the requested value.
While Binary Trie perform a variable numbers of lookup and store operations, Sparse Merkle Trie
will do constant numbers of lookup and store operations each <code>get</code> and <code>set</code> operation.</p><h3id="merkle-proofing"><ahref="#merkle-proofing"class="header-anchor">#</a> Merkle Proofing</h3><p>Using <code>prove</code> dan <code>verifyProof</code> API, we can do some merkling with SMT.</p><divclass="language-Nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">let</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br></div></div><h2id="bloom-an-ethereum-bloom-filter"><ahref="#bloom-an-ethereum-bloom-filter"class="header-anchor">#</a> bloom: an Ethereum Bloom Filter</h2><h2id="introduction-4"><ahref="#introduction-4"class="header-anchor">#</a> Introduction</h2><p>A Nim implementation of the bloom filter used by Ethereum.</p><h2id="description"><ahref="#description"class="header-anchor">#</a> Description</h2><p><ahref="https://en.wikipedia.org/wiki/Bloom_filter"target="_blank"rel="noopener noreferrer">Bloom filters<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a> are data structures that use hash functions to test whether an element is a member of a set. They work like other data structures but are probabilistic in nature: that is, they allow false positive matches but not false negative. Bloom filters use less storage space than other data structures.</p><p>Ethereum bloom filters are implemented with the Keccak-256 cryptographic hash function.</p><p>To see the bloom filter used in the context of Ethereum, please refer to the <ahref="https://ethereum.github.io/yellowpaper/paper.pdf"target="_blank"rel="noopener noreferrer">Ethereum Yellow Paper<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>.</p><h2id="usage"><ahref="#usage"class="header-anchor">#</a> Usage</h2><divclass="language-nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">import</span> eth<spanclass="token operator">/</span>bloom<spanclass="token punctuation">,</span> stint
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br></div></div><h2id="node-discovery-protocol-v5"><ahref="#node-discovery-protocol-v5"class="header-anchor">#</a> Node Discovery Protocol v5</h2><h3id="introduction-5"><ahref="#introduction-5"class="header-anchor">#</a> Introduction</h3><p>The <code>eth/p2p/discoveryv5</code> directory holds a Nim implementation of the
<ahref="https://github.com/ethereum/devp2p/blob/master/discv5/discv5.md"target="_blank"rel="noopener noreferrer">Node Discovery Protocol v5.1<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>.</p><p>The implemented specification is Protocol version v5.1.</p><p>This implementation does not support "Topic Advertisement" yet as this part of
the specification is not complete.</p><p>The implementation relies on other modules in the <code>eth</code> package, namely: <code>keys</code>,
<code>rlp</code> and <code>async_utils</code>.</p><h3id="how-to-use"><ahref="#how-to-use"class="header-anchor">#</a> How to use</h3><divclass="language-Nim line-numbers-mode"><preclass="language-nim"><code><spanclass="token keyword">let</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br></div></div><p>This will initialize the <code>Protocol</code> and start listening. However, as no
bootstrap nodes were passed in the <code>newProtocol</code> call, the created ENR will need
to be advertised somehow ("out of band"), so that the node can become known to
other nodes in the network.</p><p>To initialize with a bootnode or a set of bootnodes, the ENRs need to be passed
as parameter in <code>newProtocol</code>.</p><divclass="language-Nim line-numbers-mode"><preclass="language-nim"><code>d <spanclass="token operator">=</span><spanclass="token function">newProtocol</span><spanclass="token punctuation">(</span>privKey<spanclass="token punctuation">,</span> ip<spanclass="token punctuation">,</span> tcpPort<spanclass="token punctuation">,</span> udpPort<spanclass="token punctuation">,</span>
d<spanclass="token operator">.</span><spanclass="token function">open</span><spanclass="token punctuation">(</span><spanclass="token punctuation">)</span><spanclass="token comment"># Start listening and add bootstrap nodes to the routing table.</span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br></div></div><p>Next there are two ways to run the protocol.</p><p>One can call <code>d.start()</code> and two loops will be started:</p><ol><li>Refresh loop</li><li>Revalidation loop</li></ol><p>The first loop will at specific interval do a query with a random <code>NodeId</code> if no
manual queries were done for more than that interval period.
This query will add discovered nodes to the routing table.
The second loop will at random ranged interval send a ping to the least recently
seen node in a random bucket. This is to keep the routing table cleared of
unreachable/dead nodes.</p><p>Now within the application, manual queries or lookups can be done, for which
the discovered nodes can be used. Nodes discovered during this process will be
attempted to be added to the routing table. One can use the <code>query</code>, <code>queryRandom</code>
or <code>lookup</code> calls for this. <code>randomNodes</code> can also be used to find nodes,
but this will only look into the current routing table and not actively
search for nodes on the network.</p><p>Or, one can decide not to run <code>d.start()</code> and do this manually within its
application by using the available calls:</p><ul><li><code>query</code>, <code>queryRandom</code> or <code>lookup</code> for discovering more peers.</li><li><code>revalidateNode</code> or directly <code>ping</code> for revalidating nodes.</li></ul><p>Of course, in either scenario, lookups can still be done for actually finding a
specific node. There is a <code>resolve</code> call that can help with this, it will first
look in the local routing table and if it finds the node it will try to contact
the node directly to check if the ENR is up to date. If any of this fail a
<code>lookup</code> will be done.</p><h3id="test-suite"><ahref="#test-suite"class="header-anchor">#</a> Test suite</h3><p>To run the test suite specifically for discovery v5 related (discovery v5 + its
nim-eth dependencies) tests, one can run following command:</p><divclass="language-sh line-numbers-mode"><preclass="language-sh"><code><spanclass="token comment">## Install required modules</span>
nimble <spanclass="token function">install</span>
<spanclass="token comment">## Run only discovery v5 related test suite</span>
nimble tests_discv5_full
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br></div></div><h3id="dcli"><ahref="#dcli"class="header-anchor">#</a> dcli</h3><p>This is a small command line application that allows you to run a discovery
node. It also has the options to do a <code>ping</code> or <code>findNode</code> request to a specific
node, by providing its ENR.</p><h4id="example-usage"><ahref="#example-usage"class="header-anchor">#</a> Example usage</h4><divclass="language-sh line-numbers-mode"><preclass="language-sh"><code><spanclass="token comment">## Install required modules</span>
<spanclass="token comment">## Make sure you have the latest modules, do NOT trust nimble on this.</span>
nimble <spanclass="token function">install</span>
<spanclass="token comment">## Build dcli</span>
nim c -d:chronicles_log_level:trace -d:release --threads:on eth/p2p/discoveryv5/dcli
<spanclass="token comment">## See all options</span>
./eth/p2p/discoveryv5/dcli --help
<spanclass="token comment">## Ping another node</span>
./eth/p2p/discoveryv5/dcli <spanclass="token function">ping</span> enr:<spanclass="token operator"><</span>base64 encoding of ENR<spanclass="token operator">></span>
<spanclass="token comment">## Run discovery node</span>
./eth/p2p/discoveryv5/dcli --log-level:debug --bootnode:enr:<spanclass="token operator"><</span>base64 encoding of ENR<spanclass="token operator">></span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br><spanclass="line-number">5</span><br><spanclass="line-number">6</span><br><spanclass="line-number">7</span><br><spanclass="line-number">8</span><br><spanclass="line-number">9</span><br><spanclass="line-number">10</span><br><spanclass="line-number">11</span><br></div></div><h4id="metrics"><ahref="#metrics"class="header-anchor">#</a> Metrics</h4><p>To run dcli with metrics enabled provide the <code>metrics</code> flag:</p><divclass="language-sh line-numbers-mode"><preclass="language-sh"><code><spanclass="token comment">## Run dcli with metrics</span>
./eth/p2p/discoveryv5/dcli --metrics --bootnode:enr:<spanclass="token operator"><</span>base64 encoding of ENR<spanclass="token operator">></span>
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br></div></div><p>You can now see the metrics at http://localhost:8008/metrics. Or use e.g.
Prometheus to grab the data.</p><h2id="prerequisites"><ahref="#prerequisites"class="header-anchor">#</a> Prerequisites</h2><ul><li>Nim & Nimble</li><li>RocksDB, SQLite, LMDB (required for the trie backend tests)</li></ul><p>E.g. on Ubuntu one can run:</p><divclass="language- line-numbers-mode"><preclass="language-text"><code>apt install -y librocksdb-dev liblmdb-dev sqlite3
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br></div></div><h2id="building-testing"><ahref="#building-testing"class="header-anchor">#</a> Building & Testing</h2><divclass="language- line-numbers-mode"><preclass="language-text"><code># Install required modules
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br></div></div><p>You can also run specific parts of the test suite, e.g.:</p><divclass="language- line-numbers-mode"><preclass="language-text"><code># Test p2p functionality
nimble test_p2p
# Test rlp functionality
nimble test_rlp
</code></pre><divclass="line-numbers-wrapper"><spanclass="line-number">1</span><br><spanclass="line-number">2</span><br><spanclass="line-number">3</span><br><spanclass="line-number">4</span><br></div></div><h2id="fuzzing"><ahref="#fuzzing"class="header-anchor">#</a> Fuzzing</h2><p>Next to the test suite, there are also several fuzzing test cases available.
How these can be run is explained in the <ahref="https://github.com/status-im/nim-eth/blob/master/tests/fuzzing/readme.md"target="_blank"rel="noopener noreferrer">fuzzing readme<span><svgxmlns="http://www.w3.org/2000/svg"aria-hidden="true"focusable="false"x="0px"y="0px"viewBox="0 0 100 100"width="15"height="15"class="icon outbound"><pathfill="currentColor"d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path><polygonfill="currentColor"points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg><spanclass="sr-only">(opens new window)</span></span></a>.</p></div><footerclass="page-edit"><!----><!----></footer><!----></main></div><divclass="global-ui"></div></div>
