Initial Home page

Nimbus-Roadmap.md

@@ -0,0 +1,449 @@
+This document attempts to specify a list of high-level components of the
+Nimbus project that can be developed and tested in isolation. It tries to
+identify the dependencies between the development tasks, so the implementation
+efforts can be scheduled in parallel where appropriate. Please keep in mind
+that the list is not exhaustive and further planning is necessary before all
+tasks are specified in the same level of detail.
+
+## Development tasks:
+
+
+### 1. Select an AES implementation
+
+  The AES library should support both AES256 in CTR mode (as per [the RLPx spec][1])
+  and GCM mode (used in [Whisper v6][2])
+
+  Suggested library: Crypto++
+
+[1]: https://github.com/ethereum/devp2p/blob/master/rlpx.md
+[2]: https://gist.github.com/gluk256/cbdf8d1f05c72c2273cc42bce905388a
+
+
+### 2. Implement ECIES
+  Required for the [encyprted handshake][3]. Based on libsecp256k1.
+
+  Reference implementation:
+  - https://github.com/ethereum/py-evm/blob/master/p2p/ecies.py
+
+  Depends on:
+  * [Select an AES implementation](#select-an-aes-implementation)
+
+[3]: https://github.com/ethereum/devp2p/blob/master/rlpx.md#encrypted-handshake
+
+
+### 3. Select Scrypt library (optional)
+
+  Both cpp-ethereum and Parity also use Scrypt in their local keystore code.
+  It doesn't seem mandatory for the implementation.
+
+
+### 4. Ethereum Bloom Filter
+
+  Bloom filters are used for efficient monitoring of the transaction
+  logs/events, stored in the receipt trie of the blockchain, for topic
+  selection in Whisper and some other places.
+
+  Reference implementation:
+  - https://github.com/ethereum/eth-bloom
+
+  Existing Nim library:
+  - https://github.com/zielmicha/nim-bloom
+
+
+### 5. Ethereum key file
+  The key file stores the keys associated with one or more accounts in
+  password-protected encrypted form.
+
+  Reference implementation:
+  - https://github.com/ethereum/eth-keyfile
+
+  Sub-tasks:
+  * [ ] Safe storage
+  * [ ] Key generation / passphrase recovery
+  * [ ] Use native APIs on mobile platforms
+
+  Depends on:
+  * [Select an AES implementation](#select-an-aes-implementation)
+
+
+### 6. JSON-RPC Server
+
+  Development already started. This will be an ongoing effort and the
+  list of supported calls should increase as other parts of the system
+  are developed.
+
+  Reference information:
+  - https://github.com/ethereum/wiki/wiki/JSON-RPC
+
+
+### 7. Node discovery
+
+  - https://github.com/ethereum/devp2p/blob/master/rlpx.md#node-discovery
+
+  This uses a modified and stripped down version of [Kademlia][kad].
+  Kademlia us also used in Swarm for resource discovery and some of the
+  logic for maintaining peer connection tables can be shared between the
+  two implementations.
+
+  In the sharded blockchan, each shard is represented by a separate P2P
+  network. This is implemented by running the same node discovery protocol,
+  but with a different peer connection port:
+  - https://github.com/ethereum/py-evm/issues/196
+
+  In the future, Ethereum may support alternative transport protocols
+  (such as lib2p2). To enable this, there is new version 5 of the node
+  discovery protocol in development, which allow the nodes to advertise
+  alternative connection methods (ENR):
+
+  - https://github.com/ethereum/go-ethereum/tree/master/p2p/discv5
+  - https://github.com/fjl/EIPs/blob/f80eb12f22f178c81dc2a6de4be0c73cd302b73b/EIPS/eip-778.md
+
+  Sub-tasks:
+  - https://github.com/ethereum/devp2p/blob/master/rlpx.md#node-discovery-1
+
+[kad]: https://pdos.csail.mit.edu/~petar/papers/maymounkov-kademlia-lncs.pdf
+
+
+### 8. UPnP support
+
+  The standard Ethereum protocols use TCP to communicate between peers.
+  Hosts with private IP addresses must set up port redirection on their
+  routers. UPnP can be used to automate this.
+
+
+### 9. Authenticated handshake
+
+  https://github.com/ethereum/devp2p/blob/master/rlpx.md#encrypted-handshake
+
+  Sub-tasks:
+  * [ ] Authenticate previously known nodes with an old session key
+  * [ ] Authenticate newly discovered nodes
+  * [ ] Support RLPx sub-protocol negotiation in the authentication message (capabilities)
+  * [ ] Derive shared secrets from handshake
+ 
+  Depends on:
+  * [Select an AES implementation](#select-an-aes-implementation)
+  * [Implement ECIES](#implement-ecies)
+
+
+### 10. RLPx sub-protocols
+
+  The Ethereum wire protocols are transport-agnostic. Our framework should
+  allow running them over the  TCP connections established after node discovery
+  or through connections established with libp2p or uTP in the future.
+  It should also be possible to define unit tests involving multiple agents
+  simulated within a single process (bypassing the network stack).
+
+  **Issues:**
+
+  None of the protocols currently supports using NAT traversal for directly
+  establishing P2P connections. This may be important for supporting Video
+  and Audio calls in Status. We can solve this by establishing such connections
+  in an out-of-band protocol (e.g. by linking WebRTC or something similar) or
+  by contributing EIPs for the existing protocols.
+
+  Sub-tasks:
+
+  * [ ] SubProtocol registry
+  * [ ] Active protocol map (shared objects)
+  * [ ] Message dispatcher
+  * [ ] Subprotocol definition DSL
+
+  * [ ] Protocol negotiation
+
+  * [ ] Per-subprotocol send queue
+  * [ ] Chunking
+  * [ ] Peer reputation tracking (with persistence)
+  * [ ] Flow control (optional)
+  * [ ] Pre-sending PoW priority increase
+
+
+### 11. The new binary trie
+
+  Replaces the old hexary trie in the new blockchain shards.
+
+  Sub-tasks:
+  * [ ] Add support for partial tries
+  * [ ] Support for obtaining and verifying Merkle proofs
+
+  Reference implementations:
+  - https://github.com/ethereum/research/blob/master/trie_research/new_bintrie.py
+  - https://github.com/ethereum/py-trie/blob/master/trie/binary.py
+
+
+### 12. New parametric Blockchain definition
+
+  There should be sufficient parametrization to allow:
+
+  * [ ] The Chain is instantiated with a Trie type (either Binary or Hexary)
+  * [ ] Per-account database key handling (single layer trie)
+  * [ ] Different header structure (in the Sharded chains, header creation is also tracked in the VMC)
+  * [ ] Different transaction fields
+  * [ ] Different VM behavior (i.e. CALL opcode, the other new opcodes)
+
+  * [ ] per-transaction witness (account access lists):
+        https://github.com/ethereum/sharding/blob/develop/docs/doc.md#access-list
+        https://github.com/ethereum/py-evm/issues/195
+
+  The new shard chain uses a new account creation method:
+
+  - https://github.com/ethereum/py-evm/issues/203
+  - https://ethresear.ch/t/tradeoffs-in-account-abstraction-proposals/263
+  - https://github.com/ethereum/EIPs/blob/bd136e662fca4154787b44cded8d2a29b993be66/EIPS/abstraction.md
+
+  ... which requires 3 new opcodes to be added:
+
+  - SIGHASH
+  - PAYGAS
+  - CREATE2
+
+  Current outstanding tasks:
+  * [ ] Stateless and stateful execution with roll-back
+  * [ ] Maintenance of a receipt trie
+  * [ ] Block validation
+  * [ ] Precompiled contracts
+  * [ ] Transaction pool
+
+  Depends on:
+  * [Bloom filters](#ethereum-bloom-filter)
+  * [Partial tries](#the-new-binary-trie)
+
+  Extra tasks:
+  * VM execution tracing
+
+
+### 13. Blockchain Sync Algorithms
+
+  Fast sync uses the standard Ethereum sub-protocol.
+
+  Parity's warp sync is based on a custom server and protocol developed
+  by the Parity team.
+
+  The sharding client features an even faster stateless sync protocol
+  that aims to obtain only the latest block headers and to perform a
+  randomized verification that the blockchain data is available upon
+  request:
+
+  - https://github.com/ethereum/research/wiki/A-note-on-data-availability-and-erasure-coding
+
+  Depends on:
+  * [RLPx Sub-protocols](#rlpx-sub-protocols)
+  * Functional blockchain storage
+  * The stateless sync depends on [LES](#light-clients) and the [Binary Trie](#the-new-binary-trie)
+
+
+### 14. Known contract abstraction / DApp framework
+
+  Make it easier to interact with known contracts such as Casper,
+  the Sharding VMC, etc. Each such contract has a well-known address
+  and a particular API that can be expressed in standard Nim.
+
+  Libraries such as web3.js and web.py offer similar functionality.
+
+  Sub-tasks:
+  * [ ] Nim-types-to-ABI-types translation and contract FFI DSL.
+  * [ ] Transaction creation logic
+  * [ ] Event monitoring logic
+  * [ ] Provide high-level information about the contract for the UI layer (NatSpec, etc)
+
+  Depends on:
+  * [Bloom filters](#ethereum-bloom-filter)
+  * [RLPx Sub-protocols](#rlpx-sub-protocols)
+
+
+### 15. Light Clients
+
+  The light client operates without storing the full chain locally.
+  Most operations depends on Asynchronous APIs that will downloads
+  parts of the chain on-demand. Light clients can also execute DApps
+  by monitoring the new blocks for events of interest.
+
+  - https://github.com/ethereum/wiki/wiki/Light-client-protocol
+  - https://github.com/paritytech/parity/wiki/Light-Client
+
+  Sub-tasks:
+  * [ ] Events monitoring
+  * [ ] State-fetching APIs
+  * [ ] Flow control
+  * [ ] Fetch queue
+
+  Depends on:
+  * [Partial tries](#the-new-binary-trie)
+  * [Bloom filters](#ethereum-bloom-filter)
+  * [RLPx Sub-protocols](#rlpx-sub-protocols)
+  * DApp framework
+
+
+### 16. Stateless Clients
+
+  - https://ethresear.ch/t/the-stateless-client-concept/172
+  - https://github.com/ethereum/sharding/blob/develop/docs/doc.md#stateless-clients
+
+  Sub-tasks:
+  * [ ] Transaction state witness extraction
+  * [ ] Stateless sync
+  * [ ] State objects and "pure" VM execution
+  * [ ] Track updates to all keys in the witness in an in-memory store.
+
+  Depends on:
+  * [LES Implementation](#light-clients)
+  * [Partial tries](#the-new-binary-trie)
+
+
+### 17. Casper daemon
+
+  - https://arxiv.org/pdf/1710.09437.pdf
+  - https://github.com/ethereum/research/wiki/Casper-Version-1-Implementation-Guide
+  - https://github.com/ethereum/casper
+  - https://github.com/ethereum/wiki/wiki/Proof-of-Stake-FAQ
+
+  Sub-tasks:
+  * [ ] Implement the new Fork-choice rule
+  * [ ] Monitor epochs, send prepares and commits when appropriate.
+  * [ ] Avoid triggering the slashing conditions by maintaing a local database of past actions.
+  * [ ] Provide an UI for adding deposits, tracking revenue and withdrawing funds.
+
+  Depends on:
+  * [Known contract abstraction](#known-contract-abstraction-dapp-framework)
+
+
+### 18. Sharding daemon
+
+  - https://github.com/ethereum/sharding/blob/develop/docs/doc.md
+  - https://github.com/ethereum/wiki/wiki/Sharding-FAQ
+
+  PY-EVM Implementation Roadmap:
+  - https://github.com/ethereum/py-evm/issues/190
+
+  Sub-tasks:
+  * [ ] Shard monitoring/switching
+  * [ ] Block/Collation creation
+  * [ ] New account creation logic
+  * [ ] New VM opcodes
+  * [ ] Provide an UI for adding deposits, tracking revenue and withdrawing funds.
+
+  Depends on:
+  * [ ] Binary Trie, Overlayed/Tracked DB back-end.
+        (used to keep track of modifications created by a transaction and to model the transaction Witness data)
+
+  * [ ] Stateless Clients
+  * [ ] Parametric chain implementation
+  * [ ] [Known contract abstraction](#known-contract-abstraction-dapp-framework)
+
+
+### 19. Etehreum name service resolver
+
+  - https://docs.ens.domains/en/latest/
+
+  Sub-tasks:
+  * [ ] Resolve names by recursive ENS contract application
+
+  Depends on:
+  * [ ] Functional Blockchain repica / VM
+  * [ ] [Known contract abstraction](#known-contract-abstraction-dapp-framework)
+
+
+### 20. Swarm
+
+  - http://swarm-guide.readthedocs.io/en/latest/introduction.html
+
+  Sub-tasks:
+  * [ ] Kademlia search
+  * [ ] File chunks upload / download manager
+  * [ ] Hierarchical hash verification
+  * [ ] Virtual file system based on manifests
+  * [ ] PSS (messaging over swarm)
+  * [ ] NAT Traversal brokerage over PSS (e.g. for WebRTC)
+  * [ ] Light mode of operation, relay nodes
+  * [ ] SWAP, SWEAR, SWINDLE, chequebooks
+
+  Depends on:
+  * Reusable Kademlia routing library
+  * [ENS resolver](#ethereum-name-service-resolver)
+  * [RLPx Sub-protocols](#rlpx-sub-protocols)
+  * DApp framework
+
+
+### 21. Whisper
+
+  While Swarm seems a better fit for Status's needs, we may also implement
+  the Whisper protocol, which is currently at version 6: 
+  
+  - https://gist.github.com/gluk256/cbdf8d1f05c72c2273cc42bce905388a
+
+  Overview information (previous versions):
+
+  - https://github.com/ethereum/wiki/wiki/Whisper
+  - https://github.com/ethereum/go-ethereum/wiki/Whisper
+
+  Sub-tasks:
+  * [ ] Envelope encryption
+  * [ ] Mail Server
+  * [ ] Filter table
+  * [ ] Onion routing (Darkness)
+  
+  Depends on:
+  * [RLPx Sub-protocols](#rlpx-sub-protocols)
+  * [Bloom filters](#ethereum-bloom-filter)
+
+
+### 22. eWASM VM
+
+  Sub-tasks:
+  * [ ] Integrate Hera
+  * [ ] Extract all extern methods in a module shared between the VMs
+
+  Depends on:
+  * Parametric chain
+
+
+### 23. Integrate libsnark
+
+  Developers can take advantage of the homomorphic encryption and the
+  zero-knowledge proofs available in libsnark to implement currencies
+  and other smart contracts with stronger privacy (similar to zcash).
+  Ethereum supports this by offering the operations from libsnark as
+  precompiled contracts.
+
+  - https://blog.ethereum.org/2016/12/05/zksnarks-in-a-nutshell/
+
+  Depends on:
+  * [ ] Precompiled contracts
+
+
+### 24. Support IPFS
+
+  No concrete plans yet.
+
+
+### 25. Command-line interface, Configuration, Local directory structure, Build Targets, Web Interface
+
+  Sub-tasks:
+  * [ ] Provide options for launching the various daemons and modes of Nimbus
+  * [ ] Define a common scheme for specifying parameters (as command-line switches, as env vars, in configuration files, etc)
+  * [ ] Provide abstractions over local file system usage (e.g. XDG, AppData, Android and iOS specific APIs)
+  * [ ] Provide compile-time defines for turning off features of Nimbus (i.e. allow the lite clients to omit most of the code)
+  * [ ] Test example applications using Nimbus as a library
+  * [ ] Define various UIs for Nimbus (Web UI, GUI, etc)
+  * [ ] Android build / CI test suite
+  * [ ] iOS build / CI test suite
+
+
+### 26. Nim smart contract development
+
+  Define a Nim DSL for creating smart contracts.
+
+  Sub-tasks:
+  * [ ] Support creating library contracts
+  * [ ] Compile Nim to Solidity ASM or Vyper LLL
+  * [ ] Compile Nim to eWASM (either through the C back-end or through LLVM)
+  * [ ] Enhance security through static code analysis / formal methods
+
+
+### 27. Fuzzing framework
+
+  Test the implementation of all building block libraries such as RLP,
+  the Tries, the VM, etc through a fuzzer such as afl-fuzz. Teach the
+  fuzzer about Nim.
+
+