nim-leopard

Nim wrapper for Leopard-RS: a fast library for Reed-Solomon erasure correction coding.

Requirements

• Same as Leopard-RS' requirements, e.g. CMake 3.7 or newer.
• Nim 1.6 or newer.

Installation

With Nimble

$ nimble install leopard

In a project's .nimble file

requires "leopard >= 0.1.0 & < 0.2.0"

In a nimbus-build-system project

$ git submodule add https://github.com/status-im/nim-leopard.git vendor/nim-leopard
$ make update

Submodule

Init

status-im/leopard, a fork of catid/leopard (Leopard-RS), is a submodule of nim-leopard.

When nim-leopard is installed with nimble install leopard, or as a dependency in a Nimble project, or vendored in a nimbus-build-system project, submodule init is handled automatically.

In a standalone git clone of nim-leopard, it's necessary to init the submodule before running nimble develop or nimble install in the root of the clone

$ git submodule update --init --recursive

Build

The submodule is automatically built (in the nimcache dir) and statically linked during compilation of any Nim module that has import leopard.

If the nimcache dir is set to a custom value, it must be an absolute path.

For the build to work on Windows, nimble or nim c must be run from a Bash shell, e.g. Git Bash or an MSYS2 shell, and all needed tools (cmake, make, compiler, etc.) must be available in and suitable for that environment.

OpenMP

Leopard-RS' CMakeLists.txt checks for OpenMP support. If it is available then it is enabled in the build of libleopard.a.

Build toolchains commonly installed on Linux and Windows come with support for OpenMP.

The clang compiler that ships with Apple's Xcode does not support OpenMP, but the one installed with brew install llvm does support it, though it's also necessary to brew install libomp.

So, on macOS, when running nimble test of nim-leopard or compiling a project that imports nim-leopard:

• If libomp is not installed and Xcode clang is used, no extra flags need to be passed to the Nim compiler. OpenMP support will not be enabled in libleopard.a.
• If libomp is installed and Xcode clang is used, this flag should be passed to nim c

  -d:LeopardCmakeFlags="-DCMAKE_BUILD_TYPE=Release -DENABLE_OPENMP=off"
  

• If the intent is to use brew-installed clang + libomp, the shell environment should be modified

  $ export PATH="$(brew --prefix)/opt/llvm/bin:${PATH}"
  $ export LDFLAGS="-L$(brew --prefix)/opt/libomp/lib -L$(brew --prefix)/opt/llvm/lib -Wl,-rpath,$(brew --prefix)/opt/llvm/lib"
  

  and these flags should be passed to nim c

  -d:LeopardCmakeFlags="-DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=$(brew --prefix)/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=$(brew --prefix)/opt/llvm/bin/clang++" -d:LeopardExtraCompilerlags="-fopenmp" -d:LeopardExtraLinkerFlags="-fopenmp -L$(brew --prefix)/opt/libomp/lib"
  

Usage

import pkg/leopard

# Choose some byte and symbol counts
let
  bufSize = 64  # byte count per buffer, must be a multiple of 64
  buffers = 239 # number of data symbols
  parity = 17   # number of parity symbols

# Initialize an encoder and decoder
var
  encoderRes = LeoEncoder.init(bufSize, buffers, parity)
  decoderRes = LeoDecoder.init(bufSize, buffers, parity)

assert encoderRes.isOk
assert decoderRes.isOk

var
  encoder = encoderRes.get
  decoder = decoderRes.get

import std/random
randomize()

# Helper to generate random data
proc genData(outerLen, innerLen: int): seq[seq[byte]] =
  newSeq(result, outerLen)
  for i in 0..<outerLen:
    newSeq(result[i], innerLen)
    for j in 0..<innerLen:
      result[i][j] = rand(255).byte

var
  data = genData(buffers, bufSize) # some random data
  parityData: seq[seq[byte]]       # container for generated parity data

newSeq(parityData, parity)
for i in 0..<parity:
  newSeq(parityData[i], bufSize)

# Encode
assert encoder.encode(data, parityData).isOk

var
  holeyData = data
  holeyParityData = parityData

# Introduce up to a total of parity-count erasures in data and parityData
holeyData[9]   = @[]
holeyData[53]  = @[]
holeyData[208] = @[]
# ...
holeyParityData[1]  = @[]
holeyParityData[14] = @[]
# ...

var
  recoveredData: seq[seq[byte]] # container for recovered data

newSeq(recoveredData, buffers)
for i in 0..<buffers:
  newSeq(recoveredData[i], bufSize)

# Decode
let
  decodeRes = decoder.decode(holeyData, holeyParityData, recoveredData)

if decodeRes.isOk:
  assert holeyData != data

  # recovered data is in indices matching the erasures
  holeyData[9] = recoveredData[9]
  holeyData[53] = recoveredData[53]
  holeyData[208] = recoveredData[208]

  assert holeyData == data

else:
  # there were more than parity-count erasures
  assert $decodeRes.error == "Not enough recovery data received"

OpenMP

When OpenMP is enabled, whether or not parallel processing kicks in depends on the byte and symbol counts:

LeoEncoder.init(bufSize = 64, buffers = 239, parity = 17, ...)

Those values seem to be a lower bound for triggering parallel processing on a local machine with a 64-bit Intel processor.

Versioning

nim-leopard generally follows the master branch of status-im/leopard such that changes there will result in a version bump for this project.

Stability

nim-leopard is currently marked as experimental and may be subject to breaking changes across any version bump until it is marked as stable.

License

Wrapper License

nim-leopard is licensed and distributed under either of:

• Apache License, Version 2.0: LICENSE-APACHEv2 or https://opensource.org/licenses/Apache-2.0
• MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT

at your option. The contents of this repository may not be copied, modified, or distributed except according to those terms.

Dependency License

Leopard-RS is licensed under the BSD 3-Clause License. See their licensing page for further information.