2019-02-04 21:07:35 +00:00
nim-confutils
=============
[](https://travis-ci.org/status-im/nim-confutils)
[](https://opensource.org/licenses/Apache-2.0)
[](https://opensource.org/licenses/MIT)
2021-06-03 00:46:21 +00:00
2019-02-04 21:07:35 +00:00
## Introduction
Confutils is a library that aims to solve the configuration problem
with a holistic approach. The run-time configuration of a program
is described as a plain Nim object type from which the library
automatically derives the code for handling command-line options,
configuration files and other platform-specific sources such as the
Windows registry.
The library focuses on providing a lot of compile-time configurability
and extensibility with a strong adherence to the DRY principle.
2019-02-04 22:46:44 +00:00
Let's illustrate the API with a highly annotated example. Our configuration
might be described in a separate module looking like this:
2019-02-04 21:07:35 +00:00
```nim
# config.nim
import
confutils/defs
type
NimbusConf* = object
#
# This is our configuration type.
#
# Each field will be considered a configuration option that may appear
# on the command-line, whitin an environment variable or a configuration
# file, or elsewhere. Custom pragmas are used to annotate the fields with
# additional metadata that is used to augment the behavior of the library.
#
logLevel* {.
2019-11-11 14:42:08 +00:00
defaultValue: LogLevel.INFO
desc: "Sets the log level" }: LogLevel
2020-12-24 05:23:24 +00:00
2019-02-04 21:07:35 +00:00
#
# This program uses a CLI interface with sub-commands (similar to git).
#
# The `StartUpCommand` enum provides the list of available sub-commands,
# but since we are specifying a default value of `noCommand` , the user
# can also launch the program without entering any particular command.
# The default command will also be omitted from help messages.
#
# Please note that the `logLevel` option above will be shared by all
# sub-commands. The rest of the nested options will be relevant only
# when the designated sub-command is being invoked.
#
case cmd* {.
command
2019-11-11 14:42:08 +00:00
defaultValue: noCommand }: StartUpCommand
2019-02-04 21:07:35 +00:00
of noCommand:
dataDir* {.
2019-11-11 14:42:08 +00:00
defaultValue: getConfigDir() / "nimbus"
2019-02-04 21:07:35 +00:00
desc: "The directory where nimbus will store all blockchain data."
2019-11-11 14:42:08 +00:00
abbr: "d" }: DirPath
2019-02-04 21:07:35 +00:00
bootstrapNodes* {.
desc: "Specifies one or more bootstrap nodes to use when connecting to the network."
2019-11-11 14:42:08 +00:00
abbr: "b"
name: "bootstrap-node" }: seq[string]
2019-02-04 21:07:35 +00:00
bootstrapNodesFile* {.
2019-11-11 14:42:08 +00:00
defaultValue: ""
2019-02-04 21:07:35 +00:00
desc: "Specifies a line-delimited file of bootsrap Ethereum network addresses"
2019-11-11 14:42:08 +00:00
abbr: "f" }: InputFile
2019-02-04 21:07:35 +00:00
tcpPort* {.
2019-11-11 14:42:08 +00:00
desc: "TCP listening port" }: int
2019-02-04 21:07:35 +00:00
udpPort* {.
2019-11-11 14:42:08 +00:00
desc: "UDP listening port" }: int
2019-02-04 21:07:35 +00:00
validators* {.
required
desc: "A path to a pair of public and private keys for a validator. " &
"Nimbus will automatically add the extensions .privkey and .pubkey."
2019-11-11 14:42:08 +00:00
abbr: "v"
name: "validator" }: seq[PrivateValidatorData]
2019-02-04 21:07:35 +00:00
stateSnapshot* {.
desc: "Json file specifying a recent state snapshot"
2019-11-11 14:42:08 +00:00
abbr: "s" }: Option[BeaconState]
2020-12-24 05:23:24 +00:00
2019-02-04 21:07:35 +00:00
of createChain:
chainStartupData* {.
desc: ""
2019-11-11 14:42:08 +00:00
abbr: "c" }: ChainStartupData
2019-02-04 21:07:35 +00:00
outputStateFile* {.
desc: "Output file where to write the initial state snapshot"
2019-11-11 14:42:08 +00:00
name: "out"
abbr: "o" }: OutFilePath
2019-02-04 21:07:35 +00:00
StartUpCommand* = enum
noCommand
createChain
2020-12-24 05:23:24 +00:00
2019-02-04 21:07:35 +00:00
#
# The configuration can use user-defined types that feature custom
# command-line parsing and serialization routines.
#
PrivateValidatorData* = object
privKey*: ValidatorPrivKey
randao*: Randao
```
Then from our main module, we just need to call `confutils.load` which must be
given our configuration type as a parameter:
```nim
# main.nim
import
confutils, config
when isMainModule:
let conf = NimbusConf.load()
initDatabase conf.dataDir
```
And that's it - calling `load` with default parameters will first process any
2019-02-04 22:46:44 +00:00
[command-line options ](#handling-of-command-line-options ) and then it will
try to load any missing options from the most appropriate
2019-02-05 09:49:39 +00:00
[configuration location ](#handling-of-environment-variables-and-config-files )
2019-02-04 22:46:44 +00:00
for the platform. Diagnostic messages will be provided for many simple
configuration errors and the following help message will be produced
automatically when calling the program with `program --help` :
2019-02-04 21:07:35 +00:00
```
Usage: beacon_node [OPTIONS] < command >
The following options are supported:
--logLevel=LogLevel : Sets the log level
--dataDir=DirPath : The directory where nimbus will store all blockchain data.
--bootstrapNode=seq[string] : Specifies one or more bootstrap nodes to use when connecting to the network.
--bootstrapNodesFile=FilePath : Specifies a line-delimited file of bootsrap Ethereum network addresses
--tcpPort=int : TCP listening port
--udpPort=int : UDP listening port
--validator=seq[PrivateValidatorData] : A path to a pair of public and private keys for a validator. Nimbus will automatically add the extensions .privkey and .pubkey.
--stateSnapshot=Option[BeaconState] : Json file specifying a recent state snapshot
Available sub-commands:
beacon_node createChain
--out=OutFilePath : Output file where to write the initial state snapshot
```
For simpler CLI utilities, Confutils also provides the following convenience APIs:
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
import
confutils
cli do (validators {.
desc: "number of validators"
2019-11-11 14:42:08 +00:00
abbr: "v" }: int,
2019-02-04 21:07:35 +00:00
outputDir {.
desc: "output dir to store the generated files"
2019-11-11 14:42:08 +00:00
abbr: "o" }: OutPath,
2019-02-04 21:07:35 +00:00
startupDelay {.
2019-11-11 14:42:08 +00:00
desc: "delay in seconds before starting the simulation" } = 0):
2020-12-24 05:23:24 +00:00
2019-02-04 21:07:35 +00:00
if validators < 64:
echo "The number of validators must be greater than EPOCH_LENGTH (64)"
quit(1)
```
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
import
confutils
proc main(foo: string, bar: int) =
...
dispatch(main)
```
Under the hood, using these APIs will result in calling `load` on an anonymous
configuration type having the same fields as the supplied proc params.
Any additional arguments given as `cli(args) do ...` and `dispatch(fn, args)`
2019-11-08 16:52:13 +00:00
will be passed to `load` without modification. Please note that this requires
all parameters types to be concrete (non-generic).
2019-02-04 21:07:35 +00:00
This covers the basic usage of the library and the rest of the documentation
will describe the various ways the default behavior can be tweaked or extended.
## Configuration field pragmas
A number of pragmas defined in `confutils/defs` can be attached to the
configuration fields to control the behavior of the library.
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
template desc*(v: string) {.pragma.}
```
A description of the configuration option that will appear in the produced
help messages.
2021-09-14 09:45:01 +00:00
```nim
template longDesc*(v: string) {.pragma.}
```
A long description text that will appear below regular desc. You can use
one of {'\n', '\r'} to break it into multiple lines. But you can't use
'\p' as line break.
```text
-x, --name regular description [=defVal].
longdesc line one.
longdesc line two.
longdesc line three.
```
2019-02-04 21:07:35 +00:00
-----------------
2019-02-04 22:46:44 +00:00
```nim
2019-11-11 14:42:08 +00:00
template name*(v: string) {.pragma.}
2019-02-04 21:07:35 +00:00
```
A long name of the option.
Typically, it will have to be be specified as `--longOptionName value` .
2019-02-04 22:46:44 +00:00
See [Handling of command-line options ](#handling-of-command-line-options )
2019-02-04 21:07:35 +00:00
for more details.
-----------------
2019-02-04 22:46:44 +00:00
```nim
2019-11-11 14:42:08 +00:00
template abbr*(v: string) {.pragma.}
2019-02-04 21:07:35 +00:00
```
A short name of the option.
Typically, it will be required to be specified as `-x value` .
2019-02-04 22:46:44 +00:00
See [Handling of command-line options ](#handling-of-command-line-options )
2019-02-04 21:07:35 +00:00
for more details.
-----------------
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
template defaultValue*(v: untyped) {.pragma.}
```
The default value of the option if no value was supplied by the user.
-----------------
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
template required* {.pragma.}
```
By default, all options without default values are considered required.
An exception to this rule are all `seq[T]` or `Option[T]` options for
which the "empty" value can be considered a reasonable default. You can
also extend this behavior to other user-defined types by providing the
following overloads:
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
template hasDefault*(T: type Foo): bool = true
template default*(T: type Foo): Foo = Foo(...)
```
The `required` pragma can be applied to fields having such defaultable
types to make them required.
-----------------
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
template command* {.pragma.}
```
This must be applied to an enum field that represents a possible sub-command.
See the section on [sub-commands ](#Using-sub-commands ) for more details.
-----------------
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
template argument* {.pragma.}
```
This field represents an argument to the program. If the program expects
multiple arguments, this pragma can be applied to multiple fields or to
a single `seq[T]` field depending on the desired behavior.
2021-09-02 08:12:38 +00:00
-----------------
```nim
template separator(v: string)* {.pragma.}
```
Using this pragma, a customizable separator text will be displayed just before
this field. E.g.:
```text
Network Options: # this is a separator
-a, --opt1 desc
-b, --opt2 desc
---------------- # this is a separator too
-c, --opt3 desc
```
2019-02-04 21:07:35 +00:00
## Configuration field types
The `confutils/defs` module provides a number of types frequently used
for configuration purposes:
#### `InputFile`, `InputDir`
Confutils will validate that the file/directory exists and that it can
be read by the current user.
#### `ConfigFilePath[Format]`
A file system path pointing to a configuration file in the specific format.
The actual configuration can be loaded by calling `load(path, ConfigType)` .
When the format is `WindowsRegistry` the path should indicate a registry key.
#### `OutPath`
A valid path must be given.
--------------
Furthermore, you can extend the behavior of the library by providing
overloads such as:
2019-02-04 22:46:44 +00:00
```nim
2019-02-04 21:07:35 +00:00
proc parseCmdArg*(T: type Foo, p: TaintedString): T =
## This provides parsing and validation for fields having the `Foo` type.
## You should raise `ConfigurationError` in case of detected problems.
...
proc humaneTypeName*[T](_: type MyList[T]): string =
## The returned string will be used in the help messages produced by the
## library to describe the expected type of the configuration option.
mixin humaneTypeName
return "list of " & humaneTypeName(T)
```
For config files, Confutils can work with any format supported by the
[nim-serialization ](https://github.com/status-im/nim-serialization/ ) library
and it will use the standard serialization routines defined for the field
types in this format. Fields marked with the `command` or `argument` pragmas
will be ignored.
## Handling of command-line options
Confutils includes parsers that can mimic several traditional styles of
command line interfaces. You can select the parser being used by specifying
the `CmdParser` option when calling the configuration loading APIs.
The default parser of Confutils is called `MixedCmdParser` . It tries to follow
the [robustness principle ](https://en.wikipedia.org/wiki/Robustness_principle )
by recognizing as many styles of passing command-line switches as possible.
A prefix of `--` is used to indicate a long option name, while the `-` prefix
2019-02-05 09:49:39 +00:00
uses the short option name. Multiple short options such as `-a` , `-b` and
`-c` can be combined into a single `-abc` string. Both the long and the short
2019-02-04 21:07:35 +00:00
forms can also be prefixed with `/` in the style of Windows utilities. The
option names are matched in case-insensitive fashion and certain characters
such as `_` and `-` will be ignored. The values can be separated from the
option names with a space, colon or an equal sign. `bool` flags default to
`false` and merely including them in the command line sets them to `true` .
Other provided choices are `UnixCmdParser` , `WindowsCmdParser` and `NimCmdParser`
which are based on more strict grammars following the most established
tradition of the respective platforms. All of the discussed parsers are
defined in terms of the lower-level parametric type `CustomCmdParser` that
can be tweaked further for a more custom behavior.
Please note that the choice of `CmdParser` will also affect the formatting
of the help messages. Please see the definition of the standard [Windows][WIN_CMD]
or [Posix][POSIX_CMD] command-line help syntax for mode details.
2020-12-24 05:23:24 +00:00
[WIN_CMD]: https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/command-line-syntax-key
2019-02-04 21:07:35 +00:00
[POSIX_CMD]: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html
### Using sub-commands
As seen in the [introduction example ](#introduction ), Confutils makes it
easy to create command-line interfaces featuring sub-commands in the style
of `git` or `nimble` . The structure of the sub-command tree is encoded as
a Nim case object where the sub-command name is represented by an `enum`
field having the `command` pragma. Any nested fields will be considered
options of the particular sub-command. The top-level fields will be shared
between all sub-commands.
For each available choice of command and options, Confutils will automatically
provide a `help` command and the following additional switches:
* `-h` will print a short syntax reminder for the command
* `--help` will print a full help message (just like the `help` command)
## Handling of environment variables and config files
After parsing the command line options, the default behavior of Confutils is
to try to fill any missing options by examining the contents of the environment
2019-02-05 09:49:39 +00:00
variables plus two per-user and system-wide configuration locations derived from
the program name. If you want to use Confutils only as a command-line processor
2019-02-04 21:07:35 +00:00
or a config file parser for example, you can supply an empty/nil value to the
`cmdLine` , `envTable` or `configFileEnumerator` parameters of the `load` call.
More specifically, the `load` call supports the following parameters:
#### `cmdLine`, `envTable`
The command-line parameters and the environment table of the program.
By default, these will be obtained through Nim's `os` module.
#### `EnvValuesFormat`, `envVarsPrefix`
A nim-serialization format used to deserialize the values of environment
variables. The default format is called `CmdLineFormat` and it uses the
same `parseCmdArg` calls responsible for parsing the command-line.
The names of the environment variables are prefixed by the name of the
program by default. They are matched in case-insensitive fashion and
certain characters such as `-` and `_` are ignored.
#### `configFileEnumerator`
A function responsible for returning a sequence of `ConfigFilePath` objects.
To support heterogenous config file types, you can also return a tuple of
sequences. The default behavior of Windows is to obtain the configuration
from the Windows registry by looking at the following keys:
```
HKEY_CURRENT_USER/SOFTWARE/{appVendor}/{appName}/
HKEY_LOCAL_MACHINE/SOFTWARE/{appVendor}/{appName}/
```
On Posix systems, the default behavior is attempt to load the configuration
from the following files:
```
/$HOME/.config/{appName}.{ConfigFileFormat.extension}
2019-02-05 09:49:39 +00:00
/etc/{appName}.{ConfigFileForamt.extension}
2019-02-04 21:07:35 +00:00
```
#### `ConfigFileFormat`
A [nim-serialization ](https://github.com/status-im/nim-serialization ) format
that will be used by default by Confutils.
## Customization of the help messages
The `load` call offers few more optional parameters for modifying the
produced help messages:
#### `bannerBeforeHelp`
A copyright banner or a similar message that will appear before the
automatically generated help messages.
#### `bannerAfterHelp`
A copyright banner or a similar message that will appear after the
automatically generated help messages.
#### `version`
If you provide this parameter, Confutils will automatically respond
to the standard `--version` switch. If sub-commands are used, an
additional `version` top-level command will be inserted as well.
## Compile-time options
#### `confutils_colors`
This option controls the use of colors appearing in the help messages
produced by Confutils. Possible values are:
- `NativeColors` (used by default)
In this mode, Windows builds will produce output suitable for the console
application in older versions of Windows. On Unix-like systems, this is
equivalent to specifying `AnsiColors` .
- `AnsiColors`
Output suitable for terminals supporting the standard ANSI escape codes:
https://en.wikipedia.org/wiki/ANSI_escape_code
This includes most terminal emulators on modern Unix-like systems,
Windows console replacements such as ConEmu, and the native Console
and PowerShell applications on Windows 10.
- `None` or `NoColors`
All output will be colorless.
## Contributing
The development of Confutils is sponsored by [Status.im ](https://status.im/ )
through the use of [GitCoin ](https://gitcoin.co/ ). Please take a look at our
tracker for any issues having the [bounty][BOUNTIES] tag.
When submitting pull requests, please add test cases for any new features
or fixes and make sure `nimble test` is still able to execute the entire
test suite successfully.
[BOUNTIES]: https://github.com/status-im/nim-confutils/issues?q=is%3Aissue+is%3Aopen+label%3Abounty
## License
Licensed and distributed under either of
* MIT license: [LICENSE-MIT ](LICENSE-MIT ) or http://opensource.org/licenses/MIT
or
* Apache License, Version 2.0, ([LICENSE-APACHEv2](LICENSE-APACHEv2) or http://www.apache.org/licenses/LICENSE-2.0)
at your option. This file may not be copied, modified, or distributed except according to those terms.
