nim-leopard

Nim wrapper for Leopard-RS: a fast library for Reed-Solomon erasure correction coding.

Requirements

• Same as Leopard-RS' requirements, e.g. CMake 3.7 or newer.
• Nim 1.2 or newer.

Installation

With Nimble

$ nimble install leopard

In a project's .nimble file

requires "leopard >= 0.0.1 & < 0.0.2"

In a nimbus-build-system project

$ git submodule add https://github.com/status-im/nim-leopard.git vendor/nim-leopard
$ make update

Submodule

Init

status-im/leopard, a fork of catid/leopard (Leopard-RS), is a submodule of nim-leopard.

When nim-leopard is installed with nimble install leopard, or as a dependency in a Nimble project, or vendored in a nimbus-build-system project, submodule init is handled automatically.

In a standalone git clone of nim-leopard, it's necessary to init the submodule before running nimble develop or nimble install in the root of the clone

$ git submodule update --init

Build

The submodule is automatically built (in the nimcache dir) and statically linked during compilation of any Nim module that has import leopard or import leopard/wrapper.

If the nimcache dir is set to a custom value, it must be an absolute path.

For the build to work on Windows, nimble or nim c must be run from a Bash shell, e.g. Git Bash or an MSYS2 shell, and all needed tools (cmake, make, compiler, etc.) must be available in and suitable for that environment.

OpenMP

Leopard-RS' CMakeLists.txt checks for OpenMP support. If it is available then it is enabled in the build of libleopard.a.

Build toolchains commonly installed on Linux and Windows come with support for OpenMP.

The clang compiler that ships with Apple's Xcode does not support OpenMP, but the one installed with brew install llvm does support it, though it's also necessary to brew install libomp.

So, on macOS, when running nimble test of nim-leopard or compiling a project that imports nim-leopard:

• If libomp is not installed and Xcode clang is used, no extra flags need to be passed to the Nim compiler. OpenMP support will not be enabled in libleopard.a.
• If libomp is installed and Xcode clang is used, this flag should be passed to nim c

  -d:LeopardCmakeFlags="-DCMAKE_BUILD_TYPE=Release -DENABLE_OPENMP=off"
  

• If the intent is to use brew-installed clang + libomp, the shell environment should be modified

  $ export PATH="$(brew --prefix)/opt/llvm/bin:${PATH}"
  $ export LDFLAGS="-L$(brew --prefix)/opt/libomp/lib -L$(brew --prefix)/opt/llvm/lib -Wl,-rpath,$(brew --prefix)/opt/llvm/lib"
  

  and these flags should be passed to nim c

  -d:LeopardCmakeFlags="-DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=$(brew --prefix)/opt/llvm/bin/clang -DCMAKE_CXX_COMPILER=$(brew --prefix)/opt/llvm/bin/clang++" -d:LeopardExtraCompilerlags="-fopenmp" -d:LeopardExtraLinkerFlags="-fopenmp -L$(brew --prefix)/opt/libomp/lib"
  

Usage

import leopard

# Initialize Leopard-RS
leoInit()

var
  N: Positive
  data: seq[seq[byte]]

# RS(256,239) :: 239 data symbols, 17 parity symbols
assert RS(256,239).data == 239
assert RS(256,239).parity == 17

# Choose some N for symbolBytes
N = 1
# For RS(256,239) fill data such that
assert data.len == 239
for i in data: assert i.len == N * 64

# Encode
let
  parityData = RS(256,239).encode data

assert parityData.isOk
assert parityData.get.len == 17

# Poke up to 17 holes total in data and parityData
var
  daWithHoles = data
  paWithHoles = parityData.get

daWithHoles[9]   = @[]
daWithHoles[53]  = @[]
daWithHoles[208] = @[]
# ...
paWithHoles[1]  = @[]
paWithHoles[14] = @[]
# ...

# Decode
let
  recoveredData = RS(256,239).decode(daWithHoles, paWithHoles, (N * 64).uint)

if recoveredData.isOk:
  assert recoveredData.get == data
  assert recoveredData.get != daWithHoles
else:
  # More than 17 holes were poked
  assert recoveredData.error.code == LeopardNeedMoreData

OpenMP

When OpenMP is enabled, whether or not parallel processing kicks in depends on the symbol and byte counts. RS(256,239) with symbolBytes == 64 seems to be the lower bound for triggering parallel processing on a local machine with a 64-bit Intel processor.

Versioning

nim-leopard generally follows the upstream master branch of status-im/leopard such that changes there will result in a version bump for this project.

Stability

nim-leopard is currently marked as experimental and may be subject to breaking changes across any version bump until it is marked as stable.

License

Wrapper License

nim-leopard is licensed and distributed under either of:

• Apache License, Version 2.0: LICENSE-APACHEv2 or https://opensource.org/licenses/Apache-2.0
• MIT license: LICENSE-MIT or http://opensource.org/licenses/MIT

at your option. The contents of this repository may not be copied, modified, or distributed except according to those terms.

Dependency License

Leopard-RS is licensed under the BSD 3-Clause License. See their licensing page for further information.