nim-eth/tests/fuzzing

Fuzzing

tldr:

• Install afl.
• Create a testcase.
• Run: nim fuzz.nims afl testfolder/testcase.nim

Or

• Install libFuzzer (comes with LLVM).
• Create a testcase.
• Run: nim fuzz.nims libFuzzer testfolder/testcase.nim

Fuzzing Helpers

There are two convenience templates which will help you set up a quick fuzzing test.

These are the mandatory test block and the optional init block.

Example usage:

test:
  var rlp = rlpFromBytes(@payload.toRange)
  discard rlp.inspect()

Any unhandled Exception will result in a failure of the testcase. If certain Exceptions are to be allowed to occur within the test, they should be caught.

E.g.:

test:
  try:
    var rlp = rlpFromBytes(@payload.toRange)
    discard rlp.inspect()
  except RlpError as e:
    debug "Inspect failed", err = e.msg

Supported Fuzzers

The two templates can prepare the code for both afl and libFuzzer.

You will need to install first the fuzzer you want to use.

Install afl

# Ubuntu / Debian
sudo apt-get install afl

# Fedora
dnf install american-fuzzy-lop
# for usage with clang & clang-fast you will have to install
# american-fuzzy-lop-clang or american-fuzzy-lop-clang-fast

# Arch Linux
pacman -S afl

# NixOS
nix-env -i afl

Install libFuzzer

LibFuzzer is part of llvm and will be installed together with llvm-libs in recent versions. Installing clang should install llvm-libs.

# Ubuntu / Debian
sudo apt-get install clang

# Fedora
dnf install clang

# Arch Linux
pacman -S clang

# NixOS
nix-env -iA nixos.clang_7 nixos.llvm_7

Compiling & Starting the Fuzzer

Scripted helper

There is a nimscript helper to compile & start the fuzzer:

# for afl
nim fuzz.nims afl testcase.nim

# for libFuzzer
nim fuzz.nims libFuzzer testcase.nim

Manually with afl

Compiling

With gcc:

nim c -d:afl -d:release -d:chronicles_log_level=fatal -d:noSignalHandler --cc=gcc --gcc.exe=afl-gcc --gcc.linkerexe=afl-gcc testcase.nim

The afl define is specifically required for the init and test templates.

You typically want to fuzz in -d:release and probably also want to lower down the logging. But this is not strictly necessary.

There is also a nimscript task in config.nims for this:

nim c build_afl testcase.nim

With clang:

# afl-clang
nim c -d:afl -d:noSignalHandler --cc=clang --clang.exe=afl-clang --clang.linkerexe=afl-clang ftestcase.nim
# afl-clang-fast
nim c -d:afl -d:noSignalHandler --cc=clang --clang.exe=afl-clang-fast --clang.linkerexe=afl-clang-fast testcase.nim

Starting the Fuzzer

To start the fuzzer:

afl-fuzz -i input -o results -- ./testcase

To rerun it without losing previous results/corpus:

afl-fuzz -i - -o results -- ./testcase

To run several parallel fuzzing sessions:

# Start master fuzzer
afl-fuzz -i input -o results -M fuzzer01 -- ./testcase
# Start slaves (usually 1 per core available)
afl-fuzz -i input -o results -S fuzzer02 -- ./testcase
afl-fuzz -i input -o results -S fuzzer03 -- ./testcase
# add more if needed

When compiled with -d:afl the resulting application can also be run manually by providing it input data, e.g.:

./testcase < testfile

During debugging you might not want the testcase to generate a segmentation fault on exceptions. You can do this by rebuilding the test without the -d:afl flag. Changing to -d:debug will also help but might also change the behaviour.

Manually with libFuzzer

Compiling

nim c -d:libFuzzer -d:release -d:chronicles_log_level=fatal --noMain --cc=clang --passC="-fsanitize=fuzzer" --passL="-fsanitize=fuzzer" testcase.nim

The libFuzzer define is specifically required for the init and test templates.

You typically want to fuzz in -d:release and probably also want to lower down the logging. But this is not strictly necessary.

There is also a nimscript task in config.nims for compiling:

nim c build_libFuzzer testcase.nim

Starting the Fuzzer

Starting the fuzzer is as simple as running the compiled program:

./testcase corpus_dir -runs=1000000

To see the available options:

./testcase test=1

Parallel fuzzing on 8 cores:

./fuzz-libfuzzer -jobs=8 -workers=8

You can also use the application to verify a specific test case:

./testcase input_file

Additional notes

The init template, when used with afl, is only cosmetic. It will be run before each test block, compared to libFuzzer, where it will be run only once.

In case of using afl with alf-clang-fast you can make use of aflInit() proc and aflLoop() template.

aflInit() will allow using what is called deferred instrumentation. Basically, the forking of the process will only happen after this call, where normally it is done right before main().

aflLoop: will allow for (experimental) persistant mode. It will run the test in loop (1000 iterations) with different payloads. This is more comparable with libFuzzer.

These calls are enabled with -d:clangfast, and have to be manually added. They are currently not part of the test or init templates.