modify readme.md and CI status images
This commit is contained in:
jangko
zah
parent
29efb1a9e2
commit
bb259f39bd
174
README.md
174
README.md
|
|
@ -1,172 +1,12 @@
|
|||
# Testrunner [](https://travis-ci.org/disruptek/testutils)
|
||||
# Testutils
|
||||
|
||||
## Usage
|
||||
|
||||
Command syntax:
|
||||
|
||||
```sh
|
||||
testrunner [options] path
|
||||
Run the test(s) specified at path. Will search recursively for test files
|
||||
provided path is a directory.
|
||||
Options:
|
||||
--backends:"c cpp js objc" Run tests for specified targets
|
||||
--include:"test1 test2" Run only listed tests (space/comma seperated)
|
||||
--exclude:"test1 test2" Skip listed tests (space/comma seperated)
|
||||
--update Rewrite failed tests with new output
|
||||
--sort:"source,test" Sort the tests by program and/or test mtime
|
||||
--reverse Reverse the order of tests
|
||||
--random Shuffle the order of tests
|
||||
--help Display this help and exit
|
||||
```
|
||||
|
||||
The runner will look recursively for all `*.test` files at given path.
|
||||
|
||||
## Test file options
|
||||
|
||||
The test files follow the configuration file syntax (similar as `.ini`), see
|
||||
also [nim parsecfg module](https://nim-lang.org/docs/parsecfg.html).
|
||||
|
||||
### Required
|
||||
|
||||
- **program**: A test file should have at minimum a program name. This is the name
|
||||
of the nim source minus the `.nim` extension.
|
||||
|
||||
### Optional
|
||||
|
||||
- **max_size**: To check the maximum size of the binary, in bytes.
|
||||
- **timestamp_peg**: If you don't want to use the default timestamps, you can define
|
||||
your own timestamp peg here.
|
||||
- **compile_error**: When expecting a compilation failure, the error message that
|
||||
should be expected.
|
||||
- **error_file**: When expecting a compilation failure, the source file where the
|
||||
error should occur.
|
||||
- **os**: Space and/or comma separated list of operating systems for which the
|
||||
test should be run. Defaults to `"linux, macosx, windows"`. Tests meant for a
|
||||
different OS than the host will be marked as `SKIPPED`.
|
||||
- **--skip**: This will simply skip the test (will not be marked as failure).
|
||||
|
||||
### Forwarded Options
|
||||
Any other options or key-value pairs will be forwarded to the nim compiler.
|
||||
|
||||
A **key-value** pair will become a conditional symbol + value (`-d:SYMBOL(:VAL)`)
|
||||
for the nim compiler, e.g. for `-d:chronicles_timestamps="UnixTime"` the test
|
||||
file requires:
|
||||
```ini
|
||||
chronicles_timestamps="UnixTime"
|
||||
```
|
||||
If only a key is given, an empty value will be forwarded.
|
||||
|
||||
An **option** will be forwarded as is to the nim compiler, e.g. this can be
|
||||
added in a test file:
|
||||
```ini
|
||||
--opt:size
|
||||
```
|
||||
|
||||
### Verifying Expected Output
|
||||
|
||||
For outputs to be compared, the output string should be set to the output name
|
||||
(`stdout` or _filename_) from within an _Output_ section:
|
||||
|
||||
```ini
|
||||
[Output]
|
||||
stdout="""expected stdout output"""
|
||||
file.log="""expected file output"""
|
||||
```
|
||||
|
||||
Triple quotes can be used for multiple lines.
|
||||
|
||||
### Supplying Command-line Arguments
|
||||
|
||||
Optionally specify command-line arguments as an escaped string in the following
|
||||
syntax inside any _Output_ section:
|
||||
|
||||
```ini
|
||||
[Output]
|
||||
args = "--title \"useful title\""
|
||||
```
|
||||
|
||||
### Multiple Invocations
|
||||
|
||||
Multiple _Output_ sections denote multiple test program invocations. Any
|
||||
failure of the test program to match its expected outputs will short-circuit
|
||||
and fail the test.
|
||||
|
||||
```ini
|
||||
[Output]
|
||||
stdout = ""
|
||||
args = "--no-output"
|
||||
|
||||
[Output_newlines]
|
||||
stdout = "\n\n"
|
||||
args = "--newlines"
|
||||
```
|
||||
|
||||
### Updating Expected Outputs
|
||||
|
||||
Pass the `--update` argument to `testrunner` to rewrite any failing test with
|
||||
the new outputs of the test.
|
||||
|
||||
### Concurrent Test Execution
|
||||
|
||||
When built with threads, `testrunner` will run multiple test invocations
|
||||
defined in each test file simultaneously. You can specify `nothreads` in the
|
||||
_preamble_ to disable this behavior.
|
||||
|
||||
```ini
|
||||
nothreads = true
|
||||
|
||||
[Output_1st_serial]
|
||||
args = "--first"
|
||||
|
||||
[Output_2nd_serial]
|
||||
args = "--second"
|
||||
```
|
||||
|
||||
The failure of any test will, when possible, short-circuit all other tests
|
||||
defined in the same file.
|
||||
|
||||
### CPU Affinity
|
||||
|
||||
Specify `affinity` to clamp the first _N_ concurrent test threads to the first
|
||||
_N_ CPU cores.
|
||||
|
||||
```ini
|
||||
affinity = true
|
||||
|
||||
[Output_1st_core]
|
||||
args = "--first"
|
||||
|
||||
[Output_2nd_core]
|
||||
args = "--second"
|
||||
```
|
||||
|
||||
### Testing Alternate Backends
|
||||
|
||||
By default, `testrunner` builds tests using Nim's C backend. Specify the
|
||||
`--backends` command-line option to build and run run tests with the backends
|
||||
of your choice.
|
||||
|
||||
```sh
|
||||
$ testrunner --backends="c cpp" tests
|
||||
```
|
||||
|
||||
### Setting the Order of Tests
|
||||
|
||||
By default, `testrunner` will order test compilation and execution according
|
||||
to the modification time of the test program source. You can choose to sort
|
||||
by test program mtime, too.
|
||||
|
||||
```sh
|
||||
$ testrunner --sort:test suite/
|
||||
```
|
||||
|
||||
You can `--reverse` or `--random`ize the order of tests, too.
|
||||
|
||||
### More Examples
|
||||
|
||||
See `chonicles`, where `testutils` was born:
|
||||
- https://github.com/status-im/nim-chronicles/tree/master/tests
|
||||
Testutils now is a home to:
|
||||
|
||||
* [Testrunner](testutils/readme.md)
|
||||
[](https://travis-ci.org/status-im/nim-testutils)
|
||||
[](https://ci.appveyor.com/project/nimbus/nim-testutils/branch/master)
|
||||
* [Fuzzing](testutils/fuzzing/readme.md)
|
||||
* [Fuzzing on Windows](testutils/fuzzing/fuzzing_on_windows.md)
|
||||
|
||||
## License
|
||||
Apache2 or MIT
|
||||
|
|
|
|||
|
|
@ -0,0 +1,173 @@
|
|||
# Testrunner [](https://travis-ci.org/status-im/nim-testutils)
|
||||
[](https://ci.appveyor.com/project/nimbus/nim-testutils/branch/master)
|
||||
|
||||
## Usage
|
||||
|
||||
Command syntax:
|
||||
|
||||
```sh
|
||||
testrunner [options] path
|
||||
Run the test(s) specified at path. Will search recursively for test files
|
||||
provided path is a directory.
|
||||
Options:
|
||||
--backends:"c cpp js objc" Run tests for specified targets
|
||||
--include:"test1 test2" Run only listed tests (space/comma seperated)
|
||||
--exclude:"test1 test2" Skip listed tests (space/comma seperated)
|
||||
--update Rewrite failed tests with new output
|
||||
--sort:"source,test" Sort the tests by program and/or test mtime
|
||||
--reverse Reverse the order of tests
|
||||
--random Shuffle the order of tests
|
||||
--help Display this help and exit
|
||||
```
|
||||
|
||||
The runner will look recursively for all `*.test` files at given path.
|
||||
|
||||
## Test file options
|
||||
|
||||
The test files follow the configuration file syntax (similar as `.ini`), see
|
||||
also [nim parsecfg module](https://nim-lang.org/docs/parsecfg.html).
|
||||
|
||||
### Required
|
||||
|
||||
- **program**: A test file should have at minimum a program name. This is the name
|
||||
of the nim source minus the `.nim` extension.
|
||||
|
||||
### Optional
|
||||
|
||||
- **max_size**: To check the maximum size of the binary, in bytes.
|
||||
- **timestamp_peg**: If you don't want to use the default timestamps, you can define
|
||||
your own timestamp peg here.
|
||||
- **compile_error**: When expecting a compilation failure, the error message that
|
||||
should be expected.
|
||||
- **error_file**: When expecting a compilation failure, the source file where the
|
||||
error should occur.
|
||||
- **os**: Space and/or comma separated list of operating systems for which the
|
||||
test should be run. Defaults to `"linux, macosx, windows"`. Tests meant for a
|
||||
different OS than the host will be marked as `SKIPPED`.
|
||||
- **--skip**: This will simply skip the test (will not be marked as failure).
|
||||
|
||||
### Forwarded Options
|
||||
Any other options or key-value pairs will be forwarded to the nim compiler.
|
||||
|
||||
A **key-value** pair will become a conditional symbol + value (`-d:SYMBOL(:VAL)`)
|
||||
for the nim compiler, e.g. for `-d:chronicles_timestamps="UnixTime"` the test
|
||||
file requires:
|
||||
```ini
|
||||
chronicles_timestamps="UnixTime"
|
||||
```
|
||||
If only a key is given, an empty value will be forwarded.
|
||||
|
||||
An **option** will be forwarded as is to the nim compiler, e.g. this can be
|
||||
added in a test file:
|
||||
```ini
|
||||
--opt:size
|
||||
```
|
||||
|
||||
### Verifying Expected Output
|
||||
|
||||
For outputs to be compared, the output string should be set to the output name
|
||||
(`stdout` or _filename_) from within an _Output_ section:
|
||||
|
||||
```ini
|
||||
[Output]
|
||||
stdout="""expected stdout output"""
|
||||
file.log="""expected file output"""
|
||||
```
|
||||
|
||||
Triple quotes can be used for multiple lines.
|
||||
|
||||
### Supplying Command-line Arguments
|
||||
|
||||
Optionally specify command-line arguments as an escaped string in the following
|
||||
syntax inside any _Output_ section:
|
||||
|
||||
```ini
|
||||
[Output]
|
||||
args = "--title \"useful title\""
|
||||
```
|
||||
|
||||
### Multiple Invocations
|
||||
|
||||
Multiple _Output_ sections denote multiple test program invocations. Any
|
||||
failure of the test program to match its expected outputs will short-circuit
|
||||
and fail the test.
|
||||
|
||||
```ini
|
||||
[Output]
|
||||
stdout = ""
|
||||
args = "--no-output"
|
||||
|
||||
[Output_newlines]
|
||||
stdout = "\n\n"
|
||||
args = "--newlines"
|
||||
```
|
||||
|
||||
### Updating Expected Outputs
|
||||
|
||||
Pass the `--update` argument to `testrunner` to rewrite any failing test with
|
||||
the new outputs of the test.
|
||||
|
||||
### Concurrent Test Execution
|
||||
|
||||
When built with threads, `testrunner` will run multiple test invocations
|
||||
defined in each test file simultaneously. You can specify `nothreads` in the
|
||||
_preamble_ to disable this behavior.
|
||||
|
||||
```ini
|
||||
nothreads = true
|
||||
|
||||
[Output_1st_serial]
|
||||
args = "--first"
|
||||
|
||||
[Output_2nd_serial]
|
||||
args = "--second"
|
||||
```
|
||||
|
||||
The failure of any test will, when possible, short-circuit all other tests
|
||||
defined in the same file.
|
||||
|
||||
### CPU Affinity
|
||||
|
||||
Specify `affinity` to clamp the first _N_ concurrent test threads to the first
|
||||
_N_ CPU cores.
|
||||
|
||||
```ini
|
||||
affinity = true
|
||||
|
||||
[Output_1st_core]
|
||||
args = "--first"
|
||||
|
||||
[Output_2nd_core]
|
||||
args = "--second"
|
||||
```
|
||||
|
||||
### Testing Alternate Backends
|
||||
|
||||
By default, `testrunner` builds tests using Nim's C backend. Specify the
|
||||
`--backends` command-line option to build and run run tests with the backends
|
||||
of your choice.
|
||||
|
||||
```sh
|
||||
$ testrunner --backends="c cpp" tests
|
||||
```
|
||||
|
||||
### Setting the Order of Tests
|
||||
|
||||
By default, `testrunner` will order test compilation and execution according
|
||||
to the modification time of the test program source. You can choose to sort
|
||||
by test program mtime, too.
|
||||
|
||||
```sh
|
||||
$ testrunner --sort:test suite/
|
||||
```
|
||||
|
||||
You can `--reverse` or `--random`ize the order of tests, too.
|
||||
|
||||
### More Examples
|
||||
|
||||
See `chonicles`, where `testutils` was born:
|
||||
- https://github.com/status-im/nim-chronicles/tree/master/tests
|
||||
|
||||
|
||||
## License
|
||||
Apache2 or MIT
|
||||
Loading…
Reference in New Issue