mirror of https://github.com/status-im/consul.git
docs/nia: scheduled tasks (#11283)
* docs/nia: scheduled tasks Add basic scheduled task documentation * Add source input documentation Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> * Fix some links * Update Optional Input Variables section * Apply suggestions from code review Co-authored-by: Melissa Kam <3768460+mkam@users.noreply.github.com> * Add source input documentation Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> * updated docs with new "source input" terminology where necessary * Apply suggestions from code review applied Karl's recommendations Co-authored-by: mrspanishviking <kcardenas@hashicorp.com> * addressed code review comments - added rendered input examples to terraform-modules.mdx - added hyperlinks to terraform-modules.mdx in configuration.mdx - re-worded initial description of Services Source Input in terraform-modules.mdx * addressed code review comments - fixed spacing of regexp parameter for services source input - reworded description of source input in terraform-modules.mdx * Update from feedback - Add "currently" to clarify source input is currently only supported with schedule condition - Reword inspect mode behavaior for scheduled tasks Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Co-authored-by: Melissa Kam <3768460+mkam@users.noreply.github.com> Co-authored-by: Michael Wilkerson <mwilkerson@hashicorp.com> Co-authored-by: mrspanishviking <kcardenas@hashicorp.com>
This commit is contained in:
lornasong
GitHub
parent
8d1272a26d
commit
d0cd720ea9
|
|
@ -37,7 +37,7 @@ proxy when an instance goes unhealthy.
|
|||
## Tasks
|
||||
|
||||
A task is the action triggered by the updated data monitored in Consul. It
|
||||
takes the that dynamic service data and translates it into a call to the
|
||||
takes the dynamic service data and translates it into a call to the
|
||||
infrastructure application to configure it with the updates. It uses a driver
|
||||
to push out these updates, the initial driver being a local Terraform run. An
|
||||
example of a task is to automate a firewall security policy rule with
|
||||
|
|
@ -48,8 +48,8 @@ discovered IP addresses for a set of Consul services.
|
|||
A driver encapsulates the resources required to communicate the updates to the
|
||||
network infrastructure. The following [drivers](/docs/nia/network-drivers#terraform) are supported:
|
||||
|
||||
* Terraform driver
|
||||
* Terraform Cloud driver<EnterpriseAlert inline />
|
||||
- Terraform driver
|
||||
- Terraform Cloud driver<EnterpriseAlert inline />
|
||||
|
||||
Each driver includes a set of providers that [enables support](/docs/nia/terraform-modules) for a wide variety of infrastructure applications.
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ buffer_period {
|
|||
}
|
||||
```
|
||||
|
||||
- `buffer_period` - Configures the default buffer period for all [tasks](#task) to dampen the effects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task executions. The default is enabled to reduce the number of times downstream infrastructure is updated within a short period of time. This is useful to enable in systems that have a lot of flapping.
|
||||
- `buffer_period` - Configures the default buffer period for all dynamic [tasks](#task) to dampen the effects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task executions. The default is enabled to reduce the number of times downstream infrastructure is updated within a short period of time. This is useful to enable in systems that have a lot of flapping. Buffer periods do not apply to scheduled tasks.
|
||||
- `enabled` - (bool: true) Enable or disable buffer periods globally. Specifying `min` will also enable it.
|
||||
- `min` - (string: "5s") The minimum period of time to wait after changes are detected before triggering related tasks.
|
||||
- `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`.
|
||||
|
|
@ -130,8 +130,10 @@ task {
|
|||
- `providers` - (list[string]) Providers is the list of provider names the task is dependent on. This is used to map [Terraform provider configuration](#terraform-provider) to the task.
|
||||
- `services` - (list[string]) Required depending on [`condition`](#condition) configuration. Services is the list of logical service names or service IDs the task executes on. Consul-Terraform-Sync monitors the Consul Catalog for changes to these services and triggers the task to run. Any service value not explicitly defined by a `service` block with a matching ID is assumed to be a logical service name in the default namespace. Alternative to configuring `services`, a `condition` can be configured so that the task does not trigger on changes to services (default behavior) but instead trigger on a different condition. See [Task Condition](#task-condition) configuration for more details.
|
||||
- `source` - (string: required) Source is the location the driver uses to discover the Terraform module used for automation. The source is the module path which can be local or remote on the [Terraform Registry](https://registry.terraform.io/) or private module registry. Read more on [Terraform module source and other supported types here](https://www.terraform.io/docs/modules/sources.html).
|
||||
|
||||
- To use a private module with the [`terraform` driver](#terraform-driver), run the command [`terraform login [hostname]`](https://learn.hashicorp.com/tutorials/terraform/cloud-login) to authenticate the local Terraform CLI prior to starting Consul-Terraform-Sync.
|
||||
- To use a private module with the [`terraform_cloud` driver](#terraform-cloud-driver), no extra steps are needed.
|
||||
|
||||
```hcl
|
||||
// local module example: "./terraform-cts-hello"
|
||||
source = "<PATH>"
|
||||
|
|
@ -142,6 +144,7 @@ task {
|
|||
// private module example: "my.tfe.hostname.io/my-org/hello/cts"
|
||||
source = "<HOSTNAME>/<ORGANIZATION>/<MODULE NAME>/<PROVIDER>"
|
||||
```
|
||||
|
||||
- `variable_files` - (list[string]) Specifies list of paths to [Terraform variable definition files (`.tfvars`)](https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files). The content of these files should consist of only variable name assignments. The variable assignments must match the corresponding variable declarations made available by the Terraform module for the task.
|
||||
- Variables are loaded in the order they appear in the files. Duplicate variables are overwritten with the later value. _Unless specified by the module, configure arguments for Terraform providers using [`terraform_provider` blocks](#terraform-provider)._
|
||||
```hcl
|
||||
|
|
@ -154,11 +157,12 @@ task {
|
|||
```
|
||||
- `version` - (string) The version of the provided source the task will use. For the [Terraform driver](#terraform-driver), this is the module version. The latest version will be used as the default if omitted.
|
||||
- `working_dir` - (string) The working directory to manage generated artifacts by Consul-Terraform-Sync for this task, including Terraform configuration files. By default, a working directory is created for each task as a subdirectory in the base [`working_dir`](#working_dir), e.g. `sync-tasks/task-name`.
|
||||
- `buffer_period` - Configures the buffer period for the task to dampen the effects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task execution. The default is inherited from the top level [`buffer_period` block](#global-config-options). If configured, these values will take precedence over the global buffer period. This is useful to enable for a task that is dependent on services that have a lot of flapping.
|
||||
- `buffer_period` - Configures the buffer period for a dynamic task to dampen the effects of flapping services to downstream network devices. It defines the minimum and maximum amount of time to wait for the cluster to reach a consistent state and accumulate changes before triggering task execution. The default is inherited from the top level [`buffer_period` block](#global-config-options). If configured, these values will take precedence over the global buffer period. This is useful to enable for a task that is dependent on services that have a lot of flapping. Buffer periods do not apply to scheduled tasks.
|
||||
- `enabled` - (bool) Enable or disable buffer periods for this task. Specifying `min` will also enable it.
|
||||
- `min` - (string: "5s") The minimum period of time to wait after changes are detected before triggering related tasks.
|
||||
- `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`.
|
||||
- `condition` - (obj) The requirement that, when met, triggers Consul-Terraform-Sync to execute the task. When unconfigured, the default condition is to trigger the task on changes in the services configured in [`services`](#services). Only one `condition` may be configured per task. Consul-Terraform-Sync supports different types of conditions, which each have their own configuration options. See [Task Condition](#task-condition) configuration for full details on configuration options for each condition type.
|
||||
- `source_input` - (obj) Specifies a Consul object containing values or metadata to be provided to the Terraform Module. When source input is empty, source input will be determined by the condition or services list. Only one `source_input` may be configured per task. Consul-Terraform-Sync supports different types of source input, each source input has their own configuration options. The source input block is currently only supported with [schedule condition](#schedule-condition). See [Task Source Input](#task-source-input) configuration for full details on configuration options for each source input type.
|
||||
- `terraform_version` - (string) <EnterpriseAlert inline /> The version of Terraform to use for the Terraform Cloud workspace associated with the task. Defaults to the latest compatible version supported by the organization. This option is only available when used with the [Terraform Cloud driver](#terraform-cloud-driver); otherwise, set the version within the [Terraform driver](#terraform-driver).
|
||||
|
||||
### Task Condition
|
||||
|
|
@ -183,6 +187,7 @@ task {
|
|||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `regexp` - **(beta)** (string) Only services that have a name which matches the regular expression are used by the task. If `regexp` is configured, then [`task.services`](#services) must be omitted or empty. If both a list and a regex are needed, consider including the list as part of the regex or creating separate tasks.
|
||||
|
||||
#### Catalog-Services Condition
|
||||
|
|
@ -220,7 +225,6 @@ task {
|
|||
- `regexp` - (string) Optional if [`task.services`](/docs/nia/configuration#services) is configured. Either `regexp` or `task.services` or both must be configured. Only services that have a name which matches the regular expression are used by the task. If not provided, `regexp` will default to an exact match on `task.services` list. For example, if `task.services = ["api", "web"]`, then `regexp` will default to `^api$|^web$`. See [Task Execution: Catalog Services Condition](/docs/nia/tasks#catalog-services-condition) for more details on the relationship between `task.services` and `regexp`. Some resources for more information on regular expressions: [regular expression syntax](https://github.com/google/re2/wiki/Syntax), [try out regular expression string matching](https://golang.org/pkg/regexp/#Regexp.MatchString).
|
||||
- `source_includes_var` - (bool: false) Whether or not the module configured at [`task.source`](#source) includes the [`catalog_services` variable](/docs/nia/terraform-modules#catalog-services-variable). Please refer to the documentation of the selected module for guidance on how to configure this field. If configured inconsistently with the module, Consul-Terraform-Sync will error and exit.
|
||||
|
||||
|
||||
#### Consul KV Condition
|
||||
|
||||
A consul-kv condition block configures a task to only execute on changes to a Consul KV entry. The condition can be configured for a single Consul KV entry or for any Consul KV entries that are prefixed with a given path.
|
||||
|
|
@ -249,7 +253,91 @@ task {
|
|||
- `recurse` - (bool: false) Setting to `true` instructs Consul-Terraform-Sync to treat the path as a prefix instead of a literal match.
|
||||
- `datacenter` - (string) The datacenter of the services to query for the task. If not provided, the datacenter will default to the datacenter of the agent that Consul-Terraform-Sync queries.
|
||||
- `namespace` <EnterpriseAlert inline /> - (string) The namespace of the services to query for the task. If not provided, the namespace will be inferred from the Consul-Terraform-Sync ACL token or default to the `default` namespace.
|
||||
- `source_includes_var` - (bool: false) If set to `true`, then Consul-Terraform-Sync will include the [`consul_kv` variable](/docs/nia/terraform-modules#consul-kv-variable) as an input to the module specified in the [`task.source`](#source) field. Refer to the documentation of the selected module for guidance on how to configure this field. If configured inconsistently with the module, Consul-Terraform-Sync will error and exit.
|
||||
- `source_includes_var` - (bool: false) If set to `true`, then Consul-Terraform-Sync will include the [`consul_kv` variable](/docs/nia/terraform-modules#consul-kv-variable) as a source input. Refer to the documentation of the selected module for guidance on how to configure this field. If configured inconsistently with the module, Consul-Terraform-Sync will error and exit.
|
||||
|
||||
#### Schedule Condition
|
||||
|
||||
A scheduled task has a schedule condition block, which defines the schedule for executing the task. Unlike a dynamic task, a scheduled task does not dynamically trigger on changes in Consul.
|
||||
|
||||
Schedule tasks also rely on additional task configuration, separate from the condition block to determine the source input information to provide to the task module. See [`task.services`](#services) or [`source_input`](#source_input) block configuration for details on how to configure source input.
|
||||
|
||||
See [Task Execution: Schedule Condition](/docs/nia/tasks#schedule-condition) for more information on how tasks are triggered with schedule conditions.
|
||||
|
||||
See [Terraform Module: Source Input](/docs/nia/terraform-modules#source-input) for more information on source input options for a scheduled task.
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "scheduled_task"
|
||||
description = "execute every Monday using service information from web and db"
|
||||
services = ["web", "db"]
|
||||
source = "path/to/module"
|
||||
condition "schedule" {
|
||||
cron = "* * * * Mon"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `cron` - (string: required) The CRON expression that dictates the schedule to trigger the task. For more information on CRON expressions, see the [cronexpr parsing library](https://github.com/hashicorp/cronexpr).
|
||||
|
||||
### Task Source Input
|
||||
|
||||
You can add an optional `source_input` block to the `task` block. The `source_input` block specifies a Consul object containing values or metadata to be provided to the Terraform Module. Consul-Terraform-Sync supports the following source input types.
|
||||
|
||||
~> **The source input block is currently only supported when using a schedule condition.** Adding a `source_input` block alongside any other type of condition will result in an error. To accomplish a similar behavior with other condition blocks, use the `source_includes_var` field.
|
||||
|
||||
#### Services Source Input
|
||||
|
||||
This `services` source input object defines services registered to Consul whose metadata will be used as [services source input to the Terraform Module](/docs/nia/terraform-modules/#services-source-input). The following parameters are supported:
|
||||
|
||||
| Parameter | Description | Default | Required |
|
||||
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | ------------------------------------------------------------- |
|
||||
| `regexp` | String value matching the names of Consul services to monitor. Only services that have a name matching the regular expression are used by the task. <br/><br/> If `regexp` is configured, then [`task.services`](#services) must be omitted or empty. <br/><br/> If both a list and a regex are needed, consider including the list as part of the regex or creating separate tasks. | empty string `""` | Optional unless the `task.services` option is not configured. |
|
||||
|
||||
In the following example, the scheduled task queries all Consul services with `web` as the suffix. The metadata of matching services are provided to the Terraform module.
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "schedule_condition_task"
|
||||
description = "execute every Monday using information from service names starting with web"
|
||||
source = "path/to/module"
|
||||
source_input “services” {
|
||||
regexp = "^web.*"
|
||||
}
|
||||
condition "schedule" {
|
||||
cron = "* * * * Mon"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Consul KV Source Input
|
||||
|
||||
A Consul KV source input block defines changes to Consul KV that will be monitored. These changes will then be provided as [Consul KV source input to the Terraform Module](/docs/nia/terraform-modules/#consul-kv-source-input). The source input can be configured for a single Consul KV entry or for any Consul KV entries that are prefixed with a given path. The following parameters are supported:
|
||||
|
||||
| Parameter | Description | Default | Required |
|
||||
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | -------- |
|
||||
| `path` | String value that specifies the path of the key used by the task. The path can point to a single Consul KV entry or several entries within the path. | none | Required |
|
||||
| `recurse` | Boolean value that enables Consul-Terraform-Sync to treat the path as a prefix. If set to `false`, the path will be treated as a literal match. | `false` | Optional |
|
||||
| `datacenter` | String value specifying the name of a datacenter to query for the task. | Datacenter of the agent that Consul-Terraform-Sync queries. | Optional |
|
||||
| `namespace` | <EnterpriseAlert inline /> <br/> String value indicating the namespace of the services to query for the task. | In order of precedence: <br/> 1. Inferred from the Consul-Terraform-Sync ACL token <br/> 2. The `default` namespace. | Optional |
|
||||
|
||||
In the following example, the scheduled task queries datacenter `dc1` in the `default` namespace for changes to the value held by the key `my-key`.
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "schedule_condition_task"
|
||||
description = "execute every Monday using information from Consul KV entry my-key"
|
||||
source = "path/to/module"
|
||||
source_input "consul-kv" {
|
||||
path = "my-key"
|
||||
recurse = false
|
||||
datacenter = "dc1"
|
||||
namespace = "default"
|
||||
}
|
||||
condition "schedule" {
|
||||
cron = "* * * * Mon"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Network Drivers
|
||||
|
||||
|
|
@ -328,7 +416,7 @@ driver "terraform-cloud" {
|
|||
- `organization` - (string) The Terraform Cloud organization that hosts the managed workspaces by Consul-Terraform-Sync. Can be overridden with the `TFC_ORGANIZATION` environment variable.
|
||||
- `token` - (string) Required [Team API token](https://www.terraform.io/docs/cloud/users-teams-organizations/api-tokens.html#team-api-tokens) used for authentication with Terraform Cloud and workspace management. Only workspace permissions are needed for Consul-Terraform-Sync. The token can also be provided using the `TFC_TOKEN` environment variable.
|
||||
- We recommend creating a dedicated team and team API token to isolate automation by Consul-Terraform-Sync from other Terraform Cloud operations.
|
||||
- `workspace_prefix` - (string) Specifies a prefix to prepend to the automatically-generated workspace names used for automation. This prefix will be used by all tasks that use this driver. By default, when no prefix is configured, the workspace name will be the task name. When a prefix is configured, the workspace name will be `<workspace_prefix value><task name>`. For example, if you configure the prefix as "cts_", then a task with the name "task_firewall" will have the workspace name "cts_task_firewall".
|
||||
- `workspace_prefix` - (string) Specifies a prefix to prepend to the automatically-generated workspace names used for automation. This prefix will be used by all tasks that use this driver. By default, when no prefix is configured, the workspace name will be the task name. When a prefix is configured, the workspace name will be `<workspace_prefix value><task name>`. For example, if you configure the prefix as "cts\_", then a task with the name "task_firewall" will have the workspace name "cts_task_firewall".
|
||||
- `required_providers` - (obj: required) Declare each Terraform provider used across all tasks. This can be configured the same as how you would configure [Terraform `terraform.required_providers`](https://www.terraform.io/docs/configuration/provider-requirements.html#requiring-providers) field to specify the source and version for each provider. Consul-Terraform-Sync will process these requirements when preparing each task that uses the provider.
|
||||
- `tls` - Configure TLS to allow HTTPS connections to [Terraform Enterprise](https://www.terraform.io/docs/enterprise/install/installer.html#tls-key-amp-cert).
|
||||
- `enabled` - (bool) Enable TLS. Providing a value for any of the TLS options will enable this parameter implicitly.
|
||||
|
|
|
|||
|
|
@ -19,12 +19,22 @@ Consul-Terraform-Sync executes one or more automation tasks with the most recent
|
|||
|
||||
## Glossary
|
||||
|
||||
**Condition** - A task-level defined environmental requirement that, when met, triggers the Consul-Terraform-Sync binary to execute the related task to update network infrastructure.
|
||||
**Condition** - A task-level defined environmental requirement. When a task’s condition is met, Consul-Terraform-Sync executes that task to update network infrastructure. Depending on the condition type, the condition definition may also define and enable the source input that the task provides to the configured Terraform Module.
|
||||
|
||||
**Consul-Terraform-Sync** - [GitHub repo](https://github.com/hashicorp/consul-terraform-sync) and binary/CLI name for the project that is used to perform Network Infrastructure Automation.
|
||||
|
||||
**Dynamic Tasks** - A dynamic task is a type of task that is dynamically triggered on a change to any relevant Consul catalog values e.g. service instances, Consul KV, catalog-services. See scheduled tasks for a type of non-dynamic task.
|
||||
|
||||
-> **Note:** The terminology "tasks" used throughout the documentation refers to all types of tasks except when specifically stated otherwise.
|
||||
|
||||
**Network Drivers** - Consul-Terraform-Sync uses [network drivers](/docs/nia/network-drivers) to execute and update network infrastructure. Drivers transform Consul service-level information into downstream changes by processing and abstracting API and resource details tied to specific network infrastructure.
|
||||
|
||||
**Scheduled Tasks** - A scheduled task is a type of task that is triggered only on a schedule. It is configured with a [schedule condition](/docs/nia/configuration#schedule-condition).
|
||||
|
||||
-> **Note:** The terminology "tasks" used throughout the documentation refers to all types of tasks except when specifically stated otherwise.
|
||||
|
||||
**Source Input** - A source input defines objects that provide values or metadata to the Terraform module. See [source input](/docs/nia/terraform-modules#source-input) for the supported metadata and values. For example, a user can configure a Consul KV source input to provide KV pairs as variables to their respective Terraform Module. The source input can be included in two ways. It can be specified as a parameter in a condition using `source_includes_var` and also by using the `source_input` block.
|
||||
|
||||
**Network Infrastructure Automation (NIA)** - Enables dynamic updates to network infrastructure devices triggered when specific conditions, such as service changes and registration, are met.
|
||||
|
||||
**Tasks** - A task is the translation of dynamic service information from the Consul Catalog into network infrastructure changes downstream.
|
||||
|
|
|
|||
|
|
@ -30,9 +30,13 @@ A task can be either enabled or disabled using the [task cli](/docs/nia/cli/task
|
|||
|
||||
## Task Execution
|
||||
|
||||
An enabled task can be configured to monitor and execute on different types of conditions. For example, on changes to services ([services condition](/docs/nia/tasks#services-condition)) or on service registration and deregistration ([catalog-services condition](/docs/nia/tasks#catalog-services-condition)).
|
||||
An enabled task can be configured to monitor and execute on different types of conditions, such as changes to services ([services condition](/docs/nia/tasks#services-condition)) or service registration and deregistration ([catalog-services condition](/docs/nia/tasks#catalog-services-condition)).
|
||||
|
||||
A task can also monitor but not execute on other variables that are simply included to provide additional information to the task's module. For example, a task with a catalog-services condition may execute on registration changes but additionally monitor service instances for IP information. The details of what values are monitored and what values can execute the task are determined by the module.
|
||||
A task can also monitor, but not execute on, other variables that provide additional information to the task's module. For example, a task with a catalog-services condition may execute on registration changes, and monitor service instances for IP information.
|
||||
|
||||
A source input can be specified that implicitly includes variables to be provided to the task’s module. For example, a task can specify a Consul KV source input. The specified KV keys or key paths would be monitored for changes. Any changes detected would be included as input information for the modules. The module determines the details of what values are monitored and what values can execute the task.
|
||||
|
||||
~> **The source input block is currently only supported when using a schedule condition.** Adding a source input block alongside any other type of condition will result in an error. To accomplish a similar behaviour with other condition blocks, use the `source_includes_var` field.
|
||||
|
||||
Below are details on the types of execution conditions that Consul-Terraform-Sync supports.
|
||||
|
||||
|
|
@ -105,7 +109,8 @@ Tasks with a catalog-services condition monitor and execute on service registrat
|
|||
|
||||
The catalog-services condition operates by monitoring the [Catalog List Services API](/api-docs/catalog#list-services) and executing the task when services are added or removed in the list of registered services. Note, the task does not execute on changes to the tags of the list of services. This is similar to how changes to service instance information, mentioned above, also does not execute a task.
|
||||
|
||||
Below is an example configuration for a task that will execute when a service with a name that matches the "web.*" regular expression in datacenter "dc1" has a registration change. It additionally monitors but does not execute on service instance changes to "web-api" in datacenter "dc2".
|
||||
Below is an example configuration for a task that will execute when a service with a name that matches the "web.\*" regular expression in datacenter "dc1" has a registration change. It additionally monitors but does not execute on service instance changes to "web-api" in datacenter "dc2".
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "catalog_service_condition_task"
|
||||
|
|
@ -158,6 +163,49 @@ task {
|
|||
|
||||
If the task condition's [`source_includes_var`](/docs/nia/configuration#source_includes_var-1) field is set to `true`, then the value of the Consul KV pair(s) will be available in the [`consul_kv` input variable](/docs/nia/terraform-modules#consul-kv-variable). To use the variable, add `consul_kv` as an input variable to the module, in addition to the required `services` variable. The condition type and `source_includes_var` configuration value should be documented in the module so that users can reference them when configuring a task.
|
||||
|
||||
### Schedule Condition
|
||||
|
||||
All scheduled tasks must be configured with a schedule condition. The schedule condition sets the cadence to trigger a task with a [`cron`](/docs/nia/configuration#cron) configuration. The schedule condition block does not support parameters to configure source input. As a result, inputs must be configured separately. You can configure [`task.services`](/docs/nia/configuration#services) or a [`source_input` block](/docs/nia/configuration#source_input) to set the source input.
|
||||
|
||||
Below is an example configuration for a task that will execute every Monday, which is set by the schedule condition’s [`cron`](/docs/nia/configuration#cron) configuration. The source input is defined by the `task.services` configuration. When the task is triggered on Monday, it will retrieve the latest information on 'web' and 'db' from Consul and provide this to the module’s input variables.
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "scheduled_task"
|
||||
description = "execute every Monday using service information from web and db"
|
||||
services = ["web", "db"]
|
||||
source = "path/to/module"
|
||||
condition "schedule" {
|
||||
cron = "* * * * Mon"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Below are the available options for source input types and how to configure them:
|
||||
|
||||
- [Services source input](/docs/nia/terraform-modules/#services-source-input): configure through [`task.services`](/docs/nia/configuration#services) or [`source_input "services"`](/docs/nia/configuration#services-source-input)
|
||||
- [Consul KV source input](/docs/nia/terraform-modules/#consul-kv-source-input): configure through [`source_input "consul-kv"`](/docs/nia/configuration#consul-kv-source-input)
|
||||
|
||||
#### Running Behavior
|
||||
|
||||
Scheduled tasks generally run on schedule, but they can be triggered on demand when running Consul-Terraform-Sync in the following ways:
|
||||
|
||||
- [Long-running mode](/docs/nia/cli#long-running-mode): At the beginning of the long-running mode, Consul-Terraform-Sync first passes through a once-mode phase in which all tasks are executed once. Scheduled tasks will trigger once during this once-mode phase. This behavior also applies to tasks that are not scheduled. After once-mode has completed, scheduled tasks subsequently trigger on schedule.
|
||||
|
||||
- [Inspect mode](/docs/nia/cli#inspect-mode): When running in inspect mode, the terminal will output a plan of proposed updates that would be made if the tasks were to trigger at that moment and then exit. The outputted plan for a scheduled task is also the proposed updates that would be made if the task was triggered at that moment, even if off-schedule.
|
||||
|
||||
- [Once mode](/docs/nia/cli#once-mode): During the once mode, all tasks are only triggered one time. Scheduled tasks will execute during once mode even if not on the schedule.
|
||||
|
||||
- [Enable CLI](/docs/nia/cli/task#task-enable): When a task is enabled through the CLI, any type of task, including scheduled tasks, will be triggered at that time.
|
||||
|
||||
#### Buffer Period
|
||||
|
||||
Because scheduled tasks trigger on a configured cadence, buffer periods are disabled for scheduled tasks. Any configured `buffer_period` at the global level or task level will only apply to dynamic tasks and not scheduled ones.
|
||||
|
||||
#### Events
|
||||
|
||||
[Events](#event) are stored each time a task executes. For scheduled tasks, an event will be stored each time the task triggers on schedule regardless of if there was a change in Consul catalog.
|
||||
|
||||
## Task Automation
|
||||
|
||||
Consul-Terraform-Sync will attempt to execute each enabled task once upon startup to synchronize infrastructure with the current state of Consul. The daemon will stop and exit if any error occurs while preparing the automation environment or executing a task for the first time. This helps ensure tasks have proper configuration and are executable before the daemon transitions into running tasks in full automation as service changes are discovered over time. As a result, it is not recommended to configure a task as disabled from the start. After all tasks have successfully executed once, task failures during automation will be logged and retried or attempted again after a subsequent change.
|
||||
|
|
@ -167,17 +215,21 @@ Tasks are executed near-real time when service changes are detected. For service
|
|||
## Status Information
|
||||
|
||||
Status-related information is collected and offered via [status API](/docs/nia/api#status) to provide visibility into what and how the tasks are running. Information is offered in three-levels (lowest to highest):
|
||||
- Event data
|
||||
- Task status
|
||||
- Overall status
|
||||
|
||||
- Event data
|
||||
- Task status
|
||||
- Overall status
|
||||
|
||||
These three levels form a hierarchy where each level of data informs the one higher. The lowest-level, event data, is collected each time a task runs to update network infrastructure. This event data is then aggregated to inform individual task statuses. The count distribution of all the task statuses inform the overall status's task summary.
|
||||
|
||||
### Event
|
||||
|
||||
Each time a task's services has an update, Consul-Terraform-Sync takes a series of steps in order to update network infrastructure. This process starts with updating the task's templates to fetch new service data from Consul and ends with any post-actions after modifying network infrastructure. An event is a data structure that captures information on this process of updating network infrastructure. It stores information to help understand if the update to network infrastructure was successful or not, and it stores any errors that occurred. Because disabled tasks do not update network infrastructures, they therefore do not have store events until re-enabled.
|
||||
When a task is triggered, Consul-Terraform-Sync takes a series of steps in order to update the network infrastructure. These steps consist of fetching the latest data from Consul for the task's source inputs and then updating the network infrastructure accordingly. An event captures information across this process. It stores information to help understand if the update to network infrastructure was successful or not and any errors that may have occurred.
|
||||
|
||||
A dynamic task will store an event when it is triggered by a change in Consul. A scheduled task will store an event when it is triggered on schedule, regardless if there is a change in Consul. A disabled task does not update network infrastructures, so it will not store events until until re-enabled.
|
||||
|
||||
Sample event:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "ef202675-502f-431f-b133-ed64d15b0e0e",
|
||||
|
|
@ -199,17 +251,14 @@ For complete information on the event structure, see [events in our API document
|
|||
Each time a task runs to update network infrastructure, event data is stored for that run. 5 most recent events are stored for each task, and these stored events are used to determine task status. For example, if the most recent stored event is not successful but the others are, then the task's health status is "errored".
|
||||
|
||||
Sample task status:
|
||||
|
||||
```json
|
||||
{
|
||||
"task_name": "task_b",
|
||||
"status": "errored",
|
||||
"providers": [
|
||||
"null"
|
||||
],
|
||||
"services": [
|
||||
"web",
|
||||
],
|
||||
"events_url": "/v1/status/tasks/task_b?include=events",
|
||||
"task_name": "task_b",
|
||||
"status": "errored",
|
||||
"providers": ["null"],
|
||||
"services": ["web"],
|
||||
"events_url": "/v1/status/tasks/task_b?include=events"
|
||||
}
|
||||
```
|
||||
|
||||
|
|
@ -220,6 +269,7 @@ Task status information can be retrieved with [task status API](/docs/nia/api#ta
|
|||
Overall status returns a summary of the health statuses across all tasks. The summary is the count of tasks in each health status category.
|
||||
|
||||
Sample overall status:
|
||||
|
||||
```json
|
||||
{
|
||||
"task_summary": {
|
||||
|
|
|
|||
|
|
@ -22,9 +22,185 @@ Below are the two required elements for module compatibility with Consul-Terrafo
|
|||
|
||||
### Optional Input Variables
|
||||
|
||||
Below are additional input variables provided by Consul-Terraform-Sync that may be included in a module alongside the `services` input variable. Details for each input variable contain guidance on when to include them in a module.
|
||||
In addition to the required `services` input variable, Consul-Terraform-Sync provides additional, optional input variables to be used within your module. Support for an optional input variable requires two changes:
|
||||
|
||||
- [**`catalog_services` input variable**](#catalog-services-variable) - This variable can optionally be included in modules for the [catalog-services condition](/docs/nia/tasks#catalog-services-condition). The declaration of the `catalog_services` input variable can be included at the top of the suggested `variables.tf` file alongside the required `services` input variable and other input variables. This variable functions as the response object from the [Consul catalog list services API](/api-docs/catalog#list-services) and surfaces registered services information. It is structured as a map of lists.
|
||||
1. Updating the Terraform Module to declare the input variable in the suggested `variables.tf`
|
||||
1. Adding configuration to the Consul-Terraform-Sync task block to define the source input values that should be provided to the input variables
|
||||
|
||||
See below sections for more information on [defining source input](#source-input) and [declaring optional input variables](#how-to-create-a-compatible-terraform-module) in your Terraform module.
|
||||
|
||||
### Source Input
|
||||
|
||||
A source input allows for a task to satisfy the input requirements defined by the Terraform Module’s [input variables](https://www.terraform.io/docs/language/values/variables.html), and is configured alongside a task’s condition. Both the source input and condition define objects to be monitored, but for differing reasons. The condition defines monitored objects with criteria. When this criteria is satisfied, Consul-Terraform-Sync will then trigger a task. The source input however, defines monitored objects with the intent of providing values or metadata about these objects to the Terraform Module. The source input and condition objects can be the same, such as when `task.services` is provided without a source input block and condition block, but they can also be defined using separate blocks. In this way the source input does not need to be tied to the provided condition in order to satisfy the Terraform Module.
|
||||
|
||||
There are a few ways that a source input can be defined:
|
||||
|
||||
- [**`services` list**](/docs/nia/configuration#services) - The list of services to act as a source input.
|
||||
- **`source_includes_var` condition field** - If the condition supports this field, and it is set to true, then the condition’s objects will be used as a source input. For example, if a module is defined supporting [`catalog_services` input variable](#catalog-services-variable) then, this field can be set to true in the [catalog-services condition](/docs/nia/tasks#catalog-services-condition).
|
||||
- [**`source_input` block**](/docs/nia/configuration#source-input) - This block specifically defines a source input.
|
||||
|
||||
Multiple ways of defining a source input adds configuration flexibility, and allows for optional additional input variables to be supported by Consul-Terraform-Sync alongside the `services` input variable.
|
||||
|
||||
These optional input variables include:
|
||||
|
||||
- [**`catalog_services` variable**](#catalog-services-variable)
|
||||
- [**`consul_kv` variable**](#consul-kv-variable)
|
||||
|
||||
#### Services Source Input
|
||||
|
||||
Tasks configured with a services source input monitor for changes to services. Monitoring is either performed on a configured list of services or on any services matching a provided regex.
|
||||
|
||||
Sample rendered services input:
|
||||
|
||||
```hcl
|
||||
services = {
|
||||
"web.test-server.dc1" = {
|
||||
id = "web"
|
||||
name = "web"
|
||||
kind = ""
|
||||
address = "127.0.0.1"
|
||||
port = 80
|
||||
meta = {}
|
||||
tags = ["example"]
|
||||
namespace = ""
|
||||
status = "passing"
|
||||
node = "pm8902"
|
||||
node_id = "307625d3-a1cf-9e85-ff81-12017ca4d848"
|
||||
node_address = "127.0.0.1"
|
||||
node_datacenter = "dc1"
|
||||
node_tagged_addresses = {
|
||||
lan = "127.0.0.1"
|
||||
lan_ipv4 = "127.0.0.1"
|
||||
wan = "127.0.0.1"
|
||||
wan_ipv4 = "127.0.0.1"
|
||||
}
|
||||
node_meta = {
|
||||
consul-network-segment = ""
|
||||
}
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
In order to configure a task with the services source input, the list of services that will be used for the input must be configured in one of the following ways:
|
||||
|
||||
- the task's [`services`](/docs/nia/configuration#services)
|
||||
- a [`condition "services"` block](/docs/nia/configuration#services-condition) configured with `regexp` and `source_includes_var` set to true
|
||||
- a [`source_input "services"` block](/docs/nia/configuration#services-source-input) configured with `regexp`
|
||||
|
||||
The services source input operates by monitoring the [Health List Nodes For Service API](/api-docs/health#list-nodes-for-service) and provides the latest service information to the input variables. A complete list of service information that would be provided to the module is expanded below:
|
||||
|
||||
| Attribute | Description |
|
||||
| ----------------------- | ------------------------------------------------------------------------------------------------- |
|
||||
| `id` | A unique Consul ID for this service. The service id is unique per Consul agent. |
|
||||
| `name` | The logical name of the service. Many service instances may share the same logical service name. |
|
||||
| `address` | IP address of the service host -- if empty, node address should be used. |
|
||||
| `port` | Port number of the service |
|
||||
| `meta` | List of user-defined metadata key/value pairs for the service |
|
||||
| `tags` | List of tags for the service |
|
||||
| `namespace` | Consul Enterprise namespace of the service instance |
|
||||
| `status` | Representative status for the service instance based on an aggregate of the list of health checks |
|
||||
| `node` | Name of the Consul node on which the service is registered |
|
||||
| `node_id` | ID of the node on which the service is registered. |
|
||||
| `node_address` | The IP address of the Consul node on which the service is registered. |
|
||||
| `node_datacenter` | Data center of the Consul node on which the service is registered. |
|
||||
| `node_tagged_addresses` | List of explicit LAN and WAN IP addresses for the agent |
|
||||
| `node_meta` | List of user-defined metadata key/value pairs for the node |
|
||||
|
||||
Below is an example configuration for a task that will execute on a schedule and provide information about the services matching the `regexp` parameter to the task’s module. Note that because `regexp` is set, `task.services` is omitted, and since a schedule is being used to trigger task execution, a `condition "services"` block cannot be used.
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "services_condition_task"
|
||||
description = "execute on changes to services whose name starts with web"
|
||||
providers = ["my-provider"]
|
||||
source = "path/to/services-condition-module"
|
||||
condition "schedule" {
|
||||
cron = "* * * * Mon"
|
||||
}
|
||||
source_input "services" {
|
||||
regexp = "^web.*"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Consul KV Source Input
|
||||
|
||||
Tasks configured with a Consul KV source input monitor Consul KV for changes to KV pairs that satisfy the provided configuration. The Consul KV source input operates by monitoring the [Consul KV API](/api-docs/kv#read-key) and provides these key values to the task’s module.
|
||||
|
||||
Sample rendered consul KV input:
|
||||
|
||||
```hcl
|
||||
consul_kv = {
|
||||
"my-key" = "some value"
|
||||
}
|
||||
```
|
||||
|
||||
To configure a task with the Consul KV source input, the KVs which will be used for the input must be configured in one of the following ways:
|
||||
|
||||
- a [`condition "consul-kv"` block](/docs/nia/configuration#consul-kv-condition) configured with the `source_includes_var` set to true.
|
||||
- a [`source_input "consul-kv"` block](/docs/nia/configuration#consul-kv-source-input).
|
||||
|
||||
Below is a similar example to the one provided in the [Consul KV Condition](/docs/nia/tasks#consul-kv-condition) section. However, the difference in this example is that instead of triggering based on a change to Consul KV, this task will instead execute on a schedule. Once execution is triggered, Consul KV information is then provided to the task’s module.
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "consul_kv_schedule_task"
|
||||
description = "executes on Monday monitoring Consul KV"
|
||||
providers = ["my-provider"]
|
||||
services = ["web-api"]
|
||||
source = "path/to/consul-kv-module"
|
||||
source_input "consul-kv" {
|
||||
path = "my-key"
|
||||
recurse = true
|
||||
datacenter = "dc1"
|
||||
namespace = "default"
|
||||
}
|
||||
condition "schedule" {
|
||||
cron = "* * * * Mon"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Catalog Services Source Input
|
||||
|
||||
Tasks configured with a Catalog Services source input monitors for service and tag information provided by the [Catalog List Services API](/api-docs/catalog#list-services). The source input is a map of service names to a list of tags.
|
||||
|
||||
Sample rendered catalog-services input:
|
||||
|
||||
```hcl
|
||||
catalog_services = {
|
||||
"api" = ["prod", "staging"]
|
||||
"consul" = []
|
||||
"web" = ["blue", "green"]
|
||||
}
|
||||
```
|
||||
|
||||
To configure a task with the Catalog Services source input, the catalog services which will be used for the input must be configured in one of the following ways:
|
||||
|
||||
- a [`condition "catalog-services"` block](/docs/nia/configuration#consul-kv-condition) configured with `source_includes_var` set to true.
|
||||
|
||||
-> **Note:** Currently there is no support for a `source_input “catalog-services”` block.
|
||||
|
||||
Example of a catalog-services condition which supports source input through source_includes_var:
|
||||
|
||||
```hcl
|
||||
task {
|
||||
name = "catalog_services_condition_task"
|
||||
description = "execute on registration/deregistration of services"
|
||||
providers = ["my-provider"]
|
||||
services = ["web-api"]
|
||||
source = "path/to/catalog-services-module"
|
||||
condition "catalog-services" {
|
||||
datacenter = "dc1"
|
||||
namespace = "default"
|
||||
regexp = "web.*"
|
||||
source_includes_var = true
|
||||
node_meta {
|
||||
key = "value"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## How to Create a Compatible Terraform Module
|
||||
|
||||
|
|
@ -147,7 +323,7 @@ Network infrastructure differs vastly across teams and organizations, and the au
|
|||
4. Set reasonable default values for variables that are optional, or omit default values for variables that are required module arguments.
|
||||
5. Set the [sensitive argument](https://www.terraform.io/docs/language/values/variables.html#suppressing-values-in-cli-output) for variables that contain secret or sensitive values. When set, Terraform will redact the value from output when Terraform commands are run.
|
||||
|
||||
Terraform is an explicit configuration language and requires variables to be declared, typed, and passed explicitly through as module arguments. Consul-Terraform-Sync abstracts this by creating intermediate variables at the root level from values intended for the module. These values are configured by practitioners within the [`task` block](/docs/nia/configuration#variable_files). Value assignments are parsed to interpolate the corresponding variable declaration and are written to the appropriate Terraform files. A few assumptions are made for the intermediate variables: the variables users provide Consul-Terraform-Sync are declared and supported by the module, matching name and type.
|
||||
Terraform is an explicit configuration language and requires variables to be declared, typed, and passed explicitly through as module arguments. Consul-Terraform-Sync abstracts this by creating intermediate variables at the root level from the source input. These values are configured by practitioners within the [`task` block](/docs/nia/configuration#variable_files). Value assignments are parsed to interpolate the corresponding variable declaration and are written to the appropriate Terraform files. A few assumptions are made for the intermediate variables: the variables users provide Consul-Terraform-Sync are declared and supported by the module, matching name and type.
|
||||
|
||||
### Module Guidelines
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue