status-im
/
consul
mirror of https://github.com/status-im/consul.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #14353 from hashicorp/nia/docs-0.7.0 

Nia/docs 0.7.0
This commit is contained in:
trujillo-adam 2022-09-08 11:44:04 -07:00 committed by GitHub
parent 03df6c3ac6 001b1ed8cf
commit 02f0617b60
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 770 additions and 291 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
website/content/docs/nia/api/index.mdx
View File

@ -2,32 +2,29 @@
layout: docs
page_title: Consul-Terraform-Sync API
description: >-
How to use the Consul-Terraform-Sync API
Consul-Terraform-Sync provides an API interface that lets you query CTS instance health, check CTS instance status, and run CTS tasks.
---
# Consul-Terraform-Sync API
# Consul-Terraform-Sync API Overview
When running in [daemon mode](/docs/nia/cli#daemon-mode), Consul-Terraform-Sync (CTS) serves an HTTP API interface. Details of the available API endpoints is available in the navigation to the left.
Consul-Terraform-Sync (CTS) provides an HTTP API interface for querying CTS instances and running and managing tasks.
### Port
## Port
The API is served at the default port `8558` or a different port if set with [`port` configuration](/docs/nia/configuration#port)
### Version Prefix
## Version prefix
All API routes are prefixed with `/v1/`. This documentation is for v1 of the API, which is the only version currently.
Example: `localhost:8558/v1/status`
### Error
## Request ID
Successful API requests will receive a 2XX success status code. For other unsuccessful status codes, when possible, more details will be provided in a response body containing an error object.
Each call to a CTS API endpoint returns a `request_id` field. The field is a string generated by the API. Example:
Example: Status 400 Bad Request
```json
{
"error": {
"message": "example error message: unsupported status parameter value"
}
}
```
`"request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88"`
## Error messages
The API sends a response code in the 200 range if the call is successful. If the call is unsuccessful, the API sends an error message that includes additional information when possible. Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information.

118
website/content/docs/nia/api/status.mdx
View File

@ -6,28 +6,27 @@ description: >-
---
# Status
The `/status` endpoints share status-related information for tasks. This information is available for understanding the status of individual tasks and across tasks.
The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information).
The health status value is determined by aggregating the success or failure of the event of a task detecting changes in Consul services and then updating network infrastructure. Currently, only the 5 most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information).
If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send requests to the [`/status/cluster` endpoint path](#cluster-status) on any CTS cluster member instance to receive information about the entire cluster. Calling the `status` endpoint path (without `/cluster`), however, returns a 400 error if the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information.
## Overall Status
## Status for all tasks
This endpoint currently returns the overall status information for all tasks.
This endpoint returns the overall status information for all tasks.
| Method | Path | Produces |
| ------ | ------------------- | ------------------ |
| `GET` | `/status` | `application/json` |
### Request Parameters
Currently no request parameters are offered for the overall status API.
### Request parameters
### Response Statuses
No request parameters are available for this endpoint.
| Status | Reason |
| ------ | ---------------- |
| 200 | Successfully retrieved the status |
### Response statuses
### Response Fields
- `200`: Successfully retrieved the status.
### Response fields
| Name | Type | Description |
| ----- | ------ | ------------------ |
@ -65,7 +64,7 @@ Response:
}
```
## Task Status
## Task status
This endpoint returns the individual task status information for a single specified task or for all tasks.
@ -79,7 +78,7 @@ Task health status value is determined by the success or failure of all stored [
| ------ | ------------------- | ------------------ |
| `GET` | `/status/tasks/:task_name` | `application/json` |
### Request Parameters
### Request parameters
| Name | Required | Type | Description | Default |
| -------- | -------- | ------ | ------------------ | ------- |
@ -87,14 +86,12 @@ Task health status value is determined by the success or failure of all stored [
|`include` | Optional | string | Only accepts the value `"events"`. Use to include stored event information in response. | none
|`status` | Optional | string | Only accepts health status values `"successful"`, `"errored"`, `"critical"`, or `"unknown"`. Use to filter response by tasks that have the specified health status value. Recommend setting this parameter when requesting all tasks i.e. no `task` parameter is set. | none
### Response Statuses
### Response statuses
| Status | Reason |
| ------ | ---------------- |
| 200 | Successfully retrieved the task status |
| 404 | Task with the given name not found |
- `200`: Successfully retrieved the task status
- `404`: Task with the given name not found
### Response Fields
### Response fields
The response is a JSON map of task name to a status information structure with the following fields.
@ -126,7 +123,11 @@ Event represents the process of updating network infrastructure of a task. The d
|`config.source` | string | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** Module configured for the task.
|`config.providers` | list[string] | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** List of the providers configured for the task.
### Example: All Task Statuses
### Examples
The following examples call `status` endpoints to retrieve information about tasks.
#### All task statuses
Request:
```shell-session
@ -163,7 +164,7 @@ Response:
}
```
### Example: Individual Task Status with Events
#### Individual task status with events
Request:
```shell-session
@ -225,3 +226,78 @@ Response:
}
}
```
## Cluster status
The `/v1/status/cluster` API endpoint returns information about high-availability clusters and its members, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages.
| Method | Path | Response format |
| ------ | ----------------- | ------------------ |
| `GET` | `/status/cluster` | `application/json` |
### Request parameters
Currently no request parameters are offered for the cluster status API.
### Request statuses
- `200`: Successfully retrieved the cluster status
### Response fields
The following table describes the responses that the `status/cluster` endpoint can send.
| Field | Type | Description |
| --- | ---- | --- |
| `cluster_name` | string | Identifies the name of the cluster. |
| `members` | list[[member](#member-objects)] | Contains a list of [member objects](#member-objects). Each object in the `members` list represents a CTS instance. |
#### Member objects
The following table describes the fields available for objects in the `members` array.
| Field | Type | Description |
| --- | ---- | --- |
| `address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured on the leader instance. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. |
| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. |
| `id` | string | Indicates the service registration ID. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. |
| `leader` | boolean | Identifies the cluster leader. |
| `service_name` | string | Identifies the name of the service that the instance represents. The value is set by the `service_name` field in the [Service Registration](/docs/nia/configuration#service-registration) configuration. |
### Example
The following command calls the `/status/cluster` endpoint:
```shell-session
$ curl –request GET http://localhost:8553/v1/status/cluster
```
The following example response shows three CTS instances. The cluster leader is `cts-02` and is advertising that its address is `cts-02.example.com`:
```json
{
"cluster_name": "cluster-a",
"members": [
{
"address": "cts-02.example.com",
"healthy": true,
"id": "cts-02",
"leader": true,
"service_name": "consul-terraform-sync"
},
{
"healthy": true,
"id": "cts-03",
"leader": false,
"service_name": "consul-terraform-sync"
},
{
"healthy": true,
"id": "cts-01",
"leader": false,
"service_name": "consul-terraform-sync"
}
],
"request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88"
}
```

2
website/content/docs/nia/api/tasks.mdx
View File

@ -9,6 +9,8 @@ description: >-
The `/tasks` endpoints interact with the tasks that Consul-Terraform-Sync (CTS) is responsible for running.
If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information.
## Get Tasks
This endpoint returns information about all existing tasks.

67
website/content/docs/nia/architecture.mdx
View File

@ -2,37 +2,29 @@
layout: docs
page_title: Architecture
description: >-
Consul-Terraform-Sync Architecture
Learn about the Consul-Terraform-Sync architecture and high-level CTS components, such as the Terraform driver and tasks.
---
# Consul-Terraform-Sync Architecture
Consul-Terraform-Sync (CTS) is a service-oriented tool for managing
network infrastructure near real-time. CTS runs as a daemon
and integrates the network topology maintained by your Consul cluster with your
network infrastructure to dynamically secure and connect services.
Consul-Terraform-Sync (CTS) is a service-oriented tool for managing network infrastructure near real-time. CTS runs as a daemon and integrates the network topology maintained by your Consul cluster with your network infrastructure to dynamically secure and connect services.
## 10,000 Foot View
## CTS workflow
The following diagram shows the CTS workflow as it monitors the Consul service catalog for updates.
[![Consul-Terraform-Sync Architecture](/img/nia-highlevel-diagram.svg)](/img/nia-highlevel-diagram.svg)
The diagram shows CTS monitoring the Consul service catalog
for updates and utilizing Terraform to update the state of the infrastructure.
1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and views](#watcher-and-views).
1. CTS detects a change.
1. CTS prompts Terraform to update the state of the infrastructure.
There are two principal aspects of Sync to know about corresponding to the
lines to Consul and Terraform in the diagram above. The line to Consul
represents the Watchers monitoring the state of Consul's service catalog (and
possibly KV store) while the line to Terraform represents tasks being run to
update the infrastructure.
## Watcher and Views
## Watcher and views
CTS monitors Consul for updates utilizing Consul's [Blocking
Queries](/api-docs/features/blocking) whenever supported, falling back on
polling when not. The watcher maintains a separate thread (known internally as
a view) for each value monitored, running any tasks that depend on that watched
value whenever it's updated. Say, for example, running a task to update a
proxy when an instance goes unhealthy.
CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as *watchers*.
The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it is updated. These threads are referred to as _views_. For example, a thread may run a task to update a proxy when the watcher detects that an instance has become unhealthy .
## Tasks
@ -53,8 +45,35 @@ network infrastructure. The following [drivers](/docs/nia/network-drivers#terraf
Each driver includes a set of providers that [enables support](/docs/nia/terraform-modules) for a wide variety of infrastructure applications.
## Security Guidelines
## State storage and persistence
The [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS)
tutorial contains a checklist of best practices to secure your
CTS installation for a production environment.
The following types of state information are associated with CTS.
### Terraform state information
By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). The data persists if CTS stops and the backend is configured to a remote location.
### CTS task and event data
By default, CTS stores task and event data in memory. This data is transient and does not persist. If you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha), CTS stores the data in the Consul KV. High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops.
The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options).
## Instance compatibility checks (high availability)
If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage.
The CTS instance compatibility check reports an error if the task [module](/docs/nia/configuration#module) is configured with a local module, but the module does not exist on the CTS instance. Refer to the [Terraform documentation](https://www.terraform.io/language/modules/sources#module-sources) for additional information about module sources. Example log:
```shell-session
[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory"
```
Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information.
CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it.
CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This can happen when all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected.
## Security guidelines
We recommend following the network security guidelines described in the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. The tutorial contains a checklist of best practices to secure your CTS installation for a production environment.

73
website/content/docs/nia/cli/start.mdx
View File

@ -2,61 +2,42 @@
layout: docs
page_title: Start Command
description: >-
Consul-Terraform-Sync supports start command for starting CTS as a daemon
Run the Consul-Terraform-Sync start command to start the CTS daemon.
---
# start
The `start` command starts Consul-Terraform-Sync (CTS) as a daemon. When CTS runs as a daemon, there is no default configuration to start CTS. You must set a configuration flag `-config-file` or `-config-dir`. For example:
The `start` command starts the Consul-Terraform-Sync (CTS) daemon.
## Usage
```shell-session
$ consul-terraform-sync start -config-file=config.hcl
$ consul-terraform-sync start -config-file <instance-configuration-file> [OPTIONS]
```
To review a list of available flags, use the `-help` or `-h` flag.
## Modes
CTS can be run as a daemon in different modes.
### Long-running Mode
Flag: none
Behavior: This is the default mode in which CTS passes through a once-mode phase and then turns into a long-running process. During the once-mode phase, the daemon will exit with a non-zero status if it encounters an error. After successfully passing through once-mode phase, it will begin a long-running process in which errors are logged and exiting is not expected behavior. When the long-running process begins, CTS daemon starts serving API and command requests.
### Inspect Mode
Flag: `-inspect`
Behavior: CTS will display the proposed state changes for all tasks once and exit. No changes are applied in this mode. On encountering an error before completing, CTS will exit with a non-zero status.
Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks would update network infrastructure as expected.
---
Flag: `-inspect-task [task-name]`
Behavior: This has similar behavior as `-inspect` mode for the selected task. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode.
Usage: Useful to debug one or more tasks to confirm configuration is accurate and the selected tasks would update network infrastructure as expected.
### Once Mode
Flag: `-once`
Behavior: CTS will run all tasks once with buffer periods disabled and exit. On encountering an error before completing, CTS will exit with a non-zero status.
Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks update network infrastructure as expected.
The `-config-file` or `-config-dir` flag is required. Use the flag to specify the [CTS instance configuration file](/docs/nia/configuration) or directory containing several configuration files to start a CTS cluster.
## Options
The `start` command supports the following options:
The following table describes all of the available flags.
| Name | Required | Type | Description |Default |
| --------------- | -------- | ------- | ------------------------------- | ----------------------- |
| `-config-dir ` &nbsp; &nbsp; &nbsp;| Required when`-config-file` is not set | string | A directory to load files for configuring CTS. Configuration files require an .hcl or .json file extension in order to specify their format. This option can be specified multiple times to load different directories. | none |
| `-config-file` &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;| Required when `-config-dir` is not set | string | A file to load for configuring CTS. Configuration file requires an .hcl or .json extension in order to specify their format. This option can be specified multiple times to load different configuration files. | none |
| `-inspect` | Optional | boolean | Run CTS in Inspect mode to print the proposed state changes for all tasks, and then exit. No changes are applied in this mode. | false |
| `-inspect-task` &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; | Optional | string | Run CTS in Inspect mode to print the proposed state changes for the task, and then exit. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. | none |
| `-once` | Optional | boolean | Render templates and run tasks once. Does not start the process in long-running mode and disables buffer periods. | false |
| Name | Required | Type | Description |Default |
| --- | ---- | ---- | ---- | ---- |
| <nobr>`-config-dir`</nobr>| Required when <nobr>`-config-file`</nobr> is not set | string | Specifies a directory containing CTS instance configuration files to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one directory of files. | none |
| <nobr>`-config-file`</nobr> | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none |
| <nobr>`-inspect`</nobr> | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` |
| <nobr>`-inspect-task`</nobr> | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied. <br/>You can specify the flag multiple times to display more than one task. <br/>If an error occurs before displaying all changes, CTS exits with a non-zero status. | none |
| <nobr>`-once`</nobr> | Optional | boolean | Starts CTS in once-mode. In once-mode, CTS renders templates, runs tasks once, and disables buffer periods. Refer to [Modes](#modes) for additional information. | `false` |
| <nobr>`-reset-storage`</nobr> | Optional | boolean | <EnterpriseAlert inline /> Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader. <br/>Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | `false` |
| `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` |
## Modes
By default, CTS starts in long-running mode. The following table describes all available CTS modes.
| Name | Description |How to start |
| --- | --- | --- |
| <nobr>Long-running mode</nobr> | CTS starts in once-mode and switches to a long-running process. <p>During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.</p><p>After successfully operating in once-mode, CTS begins a long-running process.</p><p> When the long-running process begins, the CTS daemon serves API and command requests.</p><p>If an error occurs, CTS logs it and continues running.</p> | No additional flags. <br/>This is the default mode. |
| Once-mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. <p>Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.</p> | Add the `-once` flag when starting CTS. |
| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. <p>Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks will update network infrastructure as expected.</p> | Add the `-inspect` flag to verify all tasks. <p>Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.</p> |
| <nobr>High availability mode</nobr> | A long-running process that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration. <p>Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.</p> |

4
website/content/docs/nia/compatibility.mdx
View File

@ -7,7 +7,7 @@ description: >-
# Compatibility
The following tables list the Consul-Terraform-Sync (CTS) version compatibility for Consul, Terraform, and Vault.
This topic describes Consul-Terraform-Sync (CTS) cross-compatibility with Consul, Terraform, and Vault.
## Consul
@ -26,8 +26,10 @@ CTS integration with Terraform is supported for the following:
| CTS Version | Terraform CLI Version | Terraform Cloud Version | Terraform Enterprise Version |
| :------------------ | :-------------------- | :---------------------- | :--------------------------- |
| CTS Enterprise 0.7 | 0.13-1.2 | Latest | v202010-2 - Latest |
| CTS Enterprise 0.4+ | 0.13 - 1.1 | Latest | v202010-2 - Latest |
| CTS Enterprise 0.3 | 0.13 - 1.1 | N/A | v202010-2 - Latest |
| CTS OSS 0.7 | 0.13 - 1.2 | N/A | N/A |
| CTS OSS 0.3+ | 0.13 - 1.1 | N/A | N/A |
| CTS OSS 0.2 | 0.13 - 1.0 | N/A | N/A |
| CTS OSS 0.1 | 0.13 - 0.14 | N/A | N/A |

61
website/content/docs/nia/configuration.mdx
View File

@ -5,11 +5,11 @@ description: >-
Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the consul-terraform-sync daemon.
---
# Configuration Options for Consul-Terraform-Sync
# Consul-Terraform-Sync Configuration
The Consul-Terraform-Sync (CTS) daemon is configured using configuration files and supports [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) (HCL) and JSON file formats.
This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL or JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax.
## Global Config Options
## Global configurations
Top level options are reserved for configuring CTS.
@ -44,7 +44,7 @@ tls {
- `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`.
- `log_level` - (string: "INFO") The log level to use for CTS logging. This defaults to "INFO". The available log levels are "TRACE", "DEBUG", "INFO", "WARN", and "ERR".
- `port` - (int: 8558) The port for CTS to use to serve API requests.
- `id` (string: Generated ID with the format `cts-<hostname>`) The ID of the CTS instance. Used as the service ID for CTS if service registration is enabled.
- `id` (string: Generated ID with the format `cts-<hostname>`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in a high availability cluster.
- `syslog` - Specifies the syslog server for logging.
- `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging.
- `facility` - (string: "local0") Name of the syslog facility to log to.
@ -82,7 +82,7 @@ license {
| `auto_retrieval` | Optional | object | Configures the license auto-retrieval used by CTS. To learn more, review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for details | Review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for defaults. |
#### Auto-Retrieval
### Auto-retrieval
You can use the `auto_retrieval` block to configure the the automatic license retrieval in CTS. When enabled, CTS attempts to retrieve a new license from its configured Consul Enterprise backend once a day. If CTS cannot retrieve a license and the current license is reaching its expiration date, CTS attempts to retrieve a license with increased frequency, as defined by the [License Expiration Date Handling](/docs/nia/enterprise/license#license-expiration-handling).
@ -92,7 +92,7 @@ You can use the `auto_retrieval` block to configure the the automatic license re
| --------- | -------- | ---- | ----------- | ------- |
| `enabled` | Optional | string | If set to true, enables license auto-retrieval | true |
## Consul
## Consul connection
The `consul` block configures the CTS connection with a Consul agent so that CTS can perform queries for task execution. It also configures the automatic registration of CTS as a service with Consul.
@ -127,7 +127,7 @@ consul {
| `transport` | Optional | [transport](/docs/nia/configuration#transport) | Low-level network connection details ||
| `service_registration` | Optional| [service_registration](/docs/nia/configuration#service-registration) | Options for how CTS will register itself as a service with a health check to Consul. ||
##### ACL Requirements
### ACL requirements
The following table describes the ACL policies required by CTS. For more information, refer to the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync) tutorial.
| Policy | Resources |
@ -140,7 +140,7 @@ The following table describes the ACL policies required by CTS. For more informa
| `keys:write` | `consul-terraform-sync/` Only required when using Consul as the [Terraform backend](/docs/nia/configuration#backend). |
#### Auth
### Auth
Configures HTTP basic authentication for communicating with Consul.
| Parameter | Required | Type | Description | Default |
@ -149,7 +149,8 @@ Configures HTTP basic authentication for communicating with Consul.
| `username` | Optional | string | Username for authentication| none |
| `password` | Optional | string | Password for authentication| none |
#### TLS
### TLS
Configure TLS to use a secure client connection with Consul. Using HTTP/2 can solve issues related to hitting Consul's maximum connection limits, as well as improve efficiency when processing many blocking queries. This option is required for Consul-Terraform-Sync when connecting to a [Consul agent with TLS verification enabled for HTTPS connections](/docs/agent/config/config-files#verify_incoming).
If Consul is using a self-signed certificate that you have not added to the global CA chain, you can set this certificate with `ca_cert` or `ca_path`. Alternatively, you can disable SSL verification by setting `verify` to false. However, disabling verification is a potential security vulnerability.
@ -164,7 +165,7 @@ If Consul is using a self-signed certificate that you have not added to the glob
| `key` | Optional | string | The path to the PEM-encoded private key file used with the client certificate configured by `cert`. Required if Consul has `verify_incoming` set to true.<br/><br/>Can also be provided through the `CONSUL_CLIENT_KEY` environment variable. | none |
| `server_name` | Optional | string | The server name to use as the Server Name Indication (SNI) for Consul when connecting via TLS.<br/><br/>Can also be provided through the `CONSUL_TLS_SERVER_NAME` environment variable. | none |
#### Transport
### Transport
Configures the low-level network connection details to Consul.
To achieve the shortest latency between a Consul service update to a task execution, configure `max_idle_conns_per_host` equal to or greater than the number of services in automation across all tasks. This value should be lower than the configured [`http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the Consul agent.
@ -182,15 +183,15 @@ If `max_idle_conns_per_host` and the number of services in automation is greater
| `tls_handshake_timeout` | Optional | string | The amount of time to wait to complete the TLS handshake. | `10s`|
#### Service Registration
### Service registration
CTS automatically registers itself with Consul as a service with a health check, using the [`id`](/docs/nia/configuration#id) configuration as the service ID. CTS deregisters itself with Consul when CTS stops gracefully. If CTS is unable to register with Consul, then it will log the error and continue without exiting.
Service registration requires that the [Consul token](/docs/nia/configuration#consul) has an ACL policy of `service:write` for the CTS service.
| Parameter | Required | Type | Description | Default |
| --------- | -------- | ---- | ----------- | ------- |
| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul.| `true` |
| `service_name` | Optional | string | The service name for CTS. | `Consul-Terraform-Sync` |
| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:<cluster-name>` format. | `true` |
| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high-availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` |
| `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node |
| `namespace` | Optional | string | <EnterpriseAlert inline /> The namespace to register CTS in. | In order of precedence: <br/> 1. Inferred from the CTS ACL token <br/> 2. The `default` namespace. |
| `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` |
@ -213,9 +214,38 @@ The default health check is an [HTTP check](/docs/discovery/checks#http-interval
| Timeout | `2s` |
| TLSSkipVerify | `false` |
## High availability
Add a `high_availability` block to your configuration to enable CTS to run in high availability mode. Refer to [Run Consul-Terrform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. The `high_availability` block contains the following configuration items.
### High availability cluster
The `cluster` parameter contains configurations for the cluster you want to operate with high availability enabled. You can configure the following options:
| Parameter | Description| Required | Type |
| --------- | ---------- | -------- | ------|
| `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String |
| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `"consul"` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object |
#### High availability cluster storage
The `high_availability.cluster.storage` object contains the following configurations.
| Parameter | Description| Required | Type |
| --------- | ---------- | -------- | ------|
| `parent_path` | Defines a parent path in the Consul KV for CTS to store state information. Default is `consul-terraform-sync/`. CTS automatically appends the cluster name to the parent path, so the effective default directory for state information is `consul-terraform-sync/<cluster-name>`. | Optional | String |
| `namespace` | Specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String |
| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | String |
### High availability instance
The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations:
- `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value.
## Service
~> Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. See [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples.
~> **Note:** Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. Refer to [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples.
A `service` block is an optional block to explicitly define the services configured in the `task` block's `services` field (deprecated). `service` blocks do not define services configured in the `task` block's `condition "services"` or `module_input "services` blocks.
@ -241,7 +271,7 @@ service {
## Task
A `task` block configures which task to execute in automation. When the task should execute can be determined by the `service` block (deprecated) and `condition` block. The `task` block may be specified multiple times to configure multiple tasks.
A `task` block configures which task to execute in automation. Use the `condition` block to specify when the task executes. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task).
```hcl
task {
@ -624,6 +654,7 @@ driver "terraform" {
- Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html). Visit the Terraform documentation links for details on backend configuration options.
- If omitted, CTS will generate default values and use configurations from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html), which stores Terraform statefiles in the Consul KV. The [ACL token provided for Consul authentication](#consul) is used to read and write to the KV store and requires [Consul KV privileges](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifier appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`.
- The remote enhanced backend is not supported with the Terraform driver to run operations in Terraform Cloud. Use the [Terraform Cloud driver](#terraform-cloud-driver) to integrate CTS with Terraform Cloud for remote workspaces and remote operations.
- The `local` backend type is not supported with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits.
- `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the CTS log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format.
- `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the CTS daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path.
- `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=<work_dir>/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files.

129
website/content/docs/nia/installation/requirements.mdx
View File

@ -1,129 +0,0 @@
---
layout: docs
page_title: Requirements
description: >-
Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon.
---
# Prerequisites Needed to Run Consul-Terraform-Sync
The following components are required to run Consul-Terraform-Sync (CTS):
* A Terraform Provider
* A Terraform Module
* A Consul cluster running outside of the `consul-terraform-sync` daemon
Practitioners can add support for their network infrastructure through Terraform providers. Once network infrastructure support exists, practitioners can add network integrations in the form of Terraform modules.
The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driver<EnterpriseAlert inline />has [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver).
## Run a Consul Cluster
Below are several steps towards a minimum Consul setup required for running CTS.
### Install Consul
CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately.
To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install).
For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul).
### Run an Agent
The Consul agent must be running in order to dynamically update network devices. To run the local Consul agent, you can run Consul in development mode which can be started with `consul agent -dev` for simplicity. For more details on running Consul agent, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started).
When running a Consul agent with CTS in production, we suggest to keep a few considerations in mind. CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, like changes to registered services. This results in multiple long running TCP connections between CTS and the agent to poll changes for each dependency. Monitoring a high number of services may quickly hit the default Consul agent connection limits.
There are 2 ways to fix this issue. The first and recommended fix is to use HTTP/2 (requires HTTPS) to communicate between Consul-Terraform-Sync and the Consul agent. When using HTTP/2 only a single connection is made and reused for all communications. See the [Consul Configuration section](/docs/nia/configuration#consul) for more. The other option is to configure [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the agent to a reasonable value proportional to the number of services monitored by Consul-Terraform-Sync.
### Register Services
CTS monitors Consul catalog for service changes which lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent either by loading a service definition or by HTTP API request.
If you are running Consul in development mode, below is an example of registering a service by HTTP API request:
```shell-session
$ echo '{
"ID": "web",
"Name": "web",
"Address": "10.10.10.10",
"Port": 8000
}' > payload.json
$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register
```
The above example registers a service named "web" with your Consul agent. This represents a non-existent web service running at 10.10.10.10:8000. Your web service is now available for CTS to consume. You can have CTS monitor the web service to execute a task and update network device(s) by configuring "web" in [`condition "services"`](/docs/nia/configuration#services-condition) of a task block. If the web service has any non-default values, it can also be configured in `condition "services"`.
For more details on registering a service by HTTP API request, refer to the [register service API docs](/api-docs/agent/service#register-service).
For more details on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started).
### Run a Cluster
The previous steps of installing and running a single Consul agent then registering a single service is sufficient to meaningfully start running CTS.
If you would like to run a Consul cluster rather than a single agent, refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started). This will walk you through the steps of running multiple Consul agents and then joining them together into a cluster.
## Network Infrastructure (using a Terraform Provider)
CTS integrations for the Terraform driver utilizes Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver of CTS inherits the expansive collection of Terraform providers to integrate with, and with release of [Terraform 0.13](https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-13/), this extends to include providers written by the community too by using [provider source](https://www.hashicorp.com/blog/adding-provider-source-to-hashicorp-terraform/).
### Finding Terraform Providers
To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers).
### How to Create a Provider
If there is no existing Terraform provider, a new Terraform provider can be [created](https://learn.hashicorp.com/tutorials/terraform/provider-setup) and [published](https://www.terraform.io/docs/registry/providers/publishing.html). The provider can then be used within a network integration task by authoring a compatible Terraform module.
## Network Integration (using a Terraform Module)
The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module along with how it is configured within a task determines the condition under which your infrastructure is updated.
Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below.
Continue to the next page to [get started with configuring CTS and how to use Terraform providers and modules for tasks.](/docs/nia/installation/configure)
### Partner Terraform Modules
The modules listed below are available to use and are compatible with CTS.
#### A10 Networks
- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia)
#### Avi Networks
- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool)
#### AWS Application Load Balancer (ALB)
- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia)
#### Checkpoint
- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia)
#### Cisco ACI
- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia)
- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia)
#### Citrix ADC
- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia)
#### F5
- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia)
#### NS1
- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia)
#### Palo Alto Networks
- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia)
- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia)

34
website/content/docs/nia/installation/run.mdx
View File

@ -1,34 +0,0 @@
---
layout: docs
page_title: Run Consul-Terraform-Sync
description: >-
Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the `consul-terraform-sync` daemon.
---
# Run Consul-Terraform-Sync
1. Move the `consul-terraform-sync` binary to a location available on your `PATH`.
```shell-session
$ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync
```
2. Create the config.hcl file, all the options are available [here](/docs/nia/configuration).
3. Run Consul-Terraform-Sync (CTS).
```shell-session
$ consul-terraform-sync start -config-file <config.hcl>
```
4. Check status of tasks. Replace port number if configured in Step 2. See additional API endpoints [here](/docs/nia/api)
```shell-session
$ curl localhost:8558/status/tasks
```
## Other Run modes
CTS allows you to inspect your configuration before applying any change and to run in once mode, meaning that you can verify the changes are correctly applied in a test run before running it in unsupervised daemon mode.
To learn more on these options check the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial.

143
website/content/docs/nia/usage/errors-ref.mdx Normal file
View File

@ -0,0 +1,143 @@
---
layout: docs
page_title: Error Messages
description: >-
Look up Consul-Terraform-Sync error message to learn how to resolve potential issues using CTS.
---
# Error Messages
This topic explains error messages you may encounter when using Consul-Terraform-Sync (CTS).
## Example error log messages
If you configured the CTS cluster to run in [high availability mode](/docs/nia/usage/run-ha) and the the local module is missing, then the following message appears in the log:
```shell-session
[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory"
```
The resolution is to add the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information.
## Example API and CLI error messages
**Error**:
```json
{
"error": {
"message": "redirect requests to leader 'cts-01' at cts-01.example.com:8558"
}
}
```
**Conditions**:
- CTS can determine the leader.
- `high_availability.instance.address` is configured for the leader.
- The CTS instance you sent the request to is not the leader.
**Resolution**:
Redirect the request to the leader instance, for example:
```shell-session
$ curl --request GET cts-01.example.com:8558/v1/tasks
```
---
**Error**:
```json
{
"error": {
"message": "redirect requests to leader 'cts-01'"
}
}
```
**Conditions**:
* CTS can determine the leader.
* The CTS instance you sent the request to is not the leader.
* `high_availability.instance.address` is not configured.
**Resolution**:
Identify the leader instance address and redirect the request to the leader. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry:
```shell-session
[INFO] ha: acquired leadership lock: id=<ID-OF-CTS-INSTANCE>.
We recommend deploying a cluster that has three instances.
---
**Error**:
```json
{
"error": {
"message": "redirect requests to leader"
}
}
```
**Conditions**:
* The CTS instance you sent the request to is not the leader.
* The CTS is unable to determine the leader.
* Note that these conditions are rare.
**Resolution**:
Identify and send the request to the leader CTS instance. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry:
```shell-session
[INFO] ha: acquired leadership lock: id=<ID-OF-CTS-INSTANCE>
```
---
**Error**:
```json
{
"error": {
"message": "this endpoint is only available with high availability configured"
}
}
```
**Conditions**:
- You called the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) without configuring CTS for [high availability](/docs/nia/usage/run-ha).
**Resolution**:
Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha).
---
**Error**:
```json
{
"error": {
"message": "example error message: unsupported status parameter value"
}
}
```
**Conditions**:
- You sent a request to the `status` API endpoint.
- The request included an unsupported parameter value.
**Resolution**:
Send a new request and verify that all of the parameter values are correct.

134
website/content/docs/nia/usage/requirements.mdx Normal file
View File

@ -0,0 +1,134 @@
---
layout: docs
page_title: Requirements
description: >-
Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon.
---
# Requirements
The following components are required to run Consul-Terraform-Sync (CTS):
- A Terraform provider
- A Terraform module
- A Consul cluster running outside of the `consul-terraform-sync` daemon
You can add support for your network infrastructure through Terraform providers so that you can apply Terraform modules to implement network integrations.
The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driver<EnterpriseAlert inline /> has [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver).
## Run a Consul cluster
Below are several steps towards a minimum Consul setup required for running CTS.
### Install Consul
CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately.
To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install).
For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul).
### Run an agent
The Consul agent must be running in order to dynamically update network devices. Refer to the [Consul agent documentation](/docs/agent/index) for information about configuring and starting a Consul agent. For hands-on instructions about running Consul agents, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started).
When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitoring a high number of services.
To avoid reaching the limit prematurely, we recommend using HTTP/2 (requires HTTPS) to communicate between CTS and the Consul agent. When using HTTP/2, CTS establishes a single connection and reuses it for all communication. Refer to the [Consul Configuration section](/docs/nia/configuration#consul) for details.
Alternatively, you can configure the [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) option to set a maximimum number of connections to meet your needs.
### Register services
CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon is operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request.
The following HTTP API request example registers a service named `web` with your Consul agent:
```shell-session
$ echo '{
"ID": "web",
"Name": "web",
"Address": "10.10.10.10",
"Port": 8000
}' > payload.json
$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register
```
The example represents a non-existent web service running at `10.10.10.10:8000` that is now available for CTS to consume.
You can configure CTS to monitor the web service, execute a task, and update network device(s) by configuring `web` in the [`condition "services"`](/docs/nia/configuration#services-condition) task block. If the web service has any non-default values, it can also be configured in `condition "services"`.
For more details on registering a service using the HTTP API endpoint, refer to the [register service API docs](/api-docs/agent/service#register-service).
For hands-on instructions on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started).
### Run a cluster
For production environments, we recommend operating a Consul cluster rather than a single agent. Refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started) for instructions on starting multiple Consul agents and joining them into a cluster.
## Network infrastructure using a Terraform provider
CTS integrations for the Terraform driver use Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver for CTS inherits the expansive collection of Terraform providers to integrate with. You can also specify a provider `source` in the [`required_providers` configuration](https://www.terraform.io/language/providers/requirements#requiring-providers) to use providers written by the community (requires Terraform 0.13 or later).
### Finding Terraform providers
To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers).
### How to create a provider
If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registry so that you can use it within a network integration task or create a compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers:
- [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup)
- [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html).
## Network integration using a Terraform module
The Terraform module for a task in CTS is the core component of the integration. It declares which resources to use and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the conditions under which your infrastructure is updated.
Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver. You can also use a [module built by partners](#partner-terraform-modules).
Refer to [Configuration](/docs/nia/configuration) for information about configuring CTS and how to use Terraform providers and modules for tasks.
### Partner Terraform Modules
The modules listed below are available to use and are compatible with CTS.
#### A10 Networks
- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia)
#### Avi Networks
- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool)
#### AWS Application Load Balancer (ALB)
- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia)
#### Checkpoint
- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia)
#### Cisco ACI
- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia)
- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia)
#### Citrix ADC
- Create, Update, and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia)
#### F5
- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia)
#### NS1
- Create, Delete, and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia)
#### Palo Alto Networks
- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia)
- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia)

181
website/content/docs/nia/usage/run-ha.mdx Normal file
View File

@ -0,0 +1,181 @@
---
layout: docs
page_title: Run Consul-Terraform-Sync with High Availability
description: >-
Improve network automation resiliency by enabling high availability for Consul-Terraform-Sync. HA enables persistent task and event data so that CTS functions as expected during a failover event.
---
# Run Consul-Terraform-Sync with High Availability
<EnterpriseAlert>
Licenses are only required for Consul-Terraform-Sync (CTS) Enterprise
</EnterpriseAlert>
This topic describes how to run Consul-Terraform-Sync (CTS) configured for high availability. High availability is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected.
## Introduction
A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process when it is configured for high availability:
1. The CTS cluster promotes a new leader from the pool of followers in the network.
1. The new leader begins running all existing tasks in `once-mode` in order to process changes that occurred during the failover transition period. In this mode, CTS runs all existing tasks one time.
1. The new leader logs any errors that occur during `once-mode` operation and the new leader continues to monitor Consul for changes.
In a standard configuration, CTS exits if errors occur when the CTS instance runs tasks in `once-mode`. In a high availability configuration, CTS logs the errors and continues to operate without interruption.
The following diagram shows operating state when high availability is enabled. CTS Instance A is the current leader and is responsible for monitoring and running tasks:
![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.svg)
The following diagram shows the CTS cluster state after the leader stops. CTS Instance B becomes the leader responsible for monitoring and running tasks.
![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.svg)
### Failover details
- The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value.
- If failover occurs during task execution, a new leader is elected. The new leader will attempt to run all tasks once before continuing to monitor for changes.
- If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode.
- If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state.
- If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode.
Note that driver behavior is consistent whether or not CTS is running in high availability mode.
## Requirements
Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS.
* CTS Enterprise 0.7 or later
* Terraform CLI 0.13 or later
* All instances in a cluster must be in the same datacenter.
You must configure appropriate ACL permissions for your cluster. Refer to [ACL permissions](#) for details.
We recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode.
## Configuration
Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](/docs/nia/configuration#high-availability) for details about the configuration fields for the `high_availability` block.
The following example configures high availability functionality for a cluster named `cts-cluster`:
<CodeBlockConfig filename="cts-config.hcl">
```hcl
high_availability {
cluster {
name = "cts-cluster"
storage "consul" {
parent_path = "cts"
namespace = "ns"
session_ttl = "30s"
}
}
instance {
address = "cts-01.example.com"
}
}
```
</CodeBlockConfig>
### ACL permissions
The `session` and `keys` resources in your Consul environment must have `write` permissions. Refer to the [ACL documentation](/docs/security/acl) for details on how to define ACL policies.
If the `high_availability.cluster.storage.namespace` field is configured, then your ACL policy must also enable `write` permissions for the `namespace` resource.
## Start a new CTS cluster
We recommend deploying a cluster that includes three CTS instances. This is so that the cluster has one leader and two followers.
1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/configuration) for all configuration options.
1. Issue the startup command and pass the configuration file. Refer to the [`start` command reference](/docs/nia/cli/start#modes) for additional information about CTS startup modes.
```shell-session
$ consul-terraform-sync start -config-file ha-config.hcl
```
1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Only the leader of the cluster will return a successful response. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses.
```shell-session
$ curl localhost:<port>/status/tasks
```
Repeat the procedure to start the remaining instances for your cluster. We recommend using near-identical configurations for all instances in your cluster. You may not be able to use exact configurations in all cases, but starting instances with the same configuration improves consistency and reduces confusion if you need to troubleshoot errors.
## Modify an instance configuration
You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks).
1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry:
```shell-session
[INFO] ha: acquired leadership lock: id=<ID-OF-CTS-INSTANCE>
```
1. Stop one of the follower CTS instances and apply the new configuration.
1. Restart the follower instance.
1. Repeat steps 2 and 3 for other follower instances in your cluster.
1. Stop the leader instance. One of the follower instances becomes the leader.
1. Apply the new configuration to the former leader instance and restart it.
## Modify tasks
When high availability is enabled, CTS persists task and event data. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information.
You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes because inconsistencies between the state and the configuration can occur when mixing methods.
### Delete and recreate the task (recommended)
Use the CTS API to identify the CTS leader instance as well as delete and replace a task.
1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry:
```shell-session
[INFO] ha: acquired leadership lock: id=<ID-OF-CTS-INSTANCE>
```
1. Send a `DELETE` call to the [`/task/<task-name>` endpoint](/docs/nia/api/tasks#delete-task) to delete the task. In the following example, the leader instance is at `localhost:8558`:
```shell-session
$ curl --request DELETE localhost:8558/v1/tasks/task_a
```
You can also use the [`task delete` command](/docs/nia/cli/task#task-delete) to complete this step.
1. Send a `POST` call to the `/task/<task-name>` endpoint and include the updated task in your payload.
```shell-session
$curl --header "Content-Type: application/json" \
--request POST \
--data @payload.json \
localhost:8558/v1/tasks
```
You can also use the [`task-create` command](/docs/nia/cli/task#task-create) to complete this step.
### Discard data with the `-reset-storage` flag
You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/start#options) to discard persisted data if you need to update a task.
1. Stop a follower instance.
1. Update the instance’s task configuration.
1. Restart the instance and include the `-reset-storage` flag.
1. Stop all other instances so that the updated instance becomes the leader.
1. Start all other instances again.
1. Restart the instance you restarted in step 3 without the `-reset-storage` flag so that it starts up with the current state. If you continue to run an instance with the `-reset-storage` flag enabled, then CTS will reset the state data whenever the instance becomes the leader.
## Troubleshooting
Use the following troubleshooting procedure if a previous leader had been running a task successfully but the new leader logs an error after a failover:
1. Check the logs printed to the console for errors. Refer to the [`syslog` configuration](/docs/nia/configuration#syslog) for information on how to locate the logs. In the following example output, CTS reported a `401: Bad credentials` error:
```shell-session
2022-08-23T09:25:09.501-0700 [ERROR] tasksmanager: error applying task: task_name=config-task
error=
| error tf-apply for 'config-task': exit status 1
|
| Error: GET https://api.github.com/user: 401 Bad credentials []
|
| with module.config-task.provider["registry.terraform.io/integrations/github"],
| on .terraform/modules/config-task/main.tf line 11, in provider "github":
| 11: provider "github" {
|
```
1. Check for differences between the previous leader and new leader, such as differences in configurations, environment variables, and local resources.
1. Start a new instance with the fix that resolves the issue.
1. Tear down the leader instance that has the issue and any other instances that may have the same issue.
1. Restart the affected instances to implement the fix.

41
website/content/docs/nia/usage/run.mdx Normal file
View File

@ -0,0 +1,41 @@
---
layout: docs
page_title: Run Consul-Terraform-Sync
description: >-
Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the `consul-terraform-sync` daemon.
---
# Run Consul-Terraform-Sync
This topic describes the basic procedure for running Consul-Terraform-Sync (CTS). Verify that you have met the [basic requirements](/docs/nia/usage/requirements) before attempting to run CTS.
1. Move the `consul-terraform-sync` binary to a location available on your `PATH`.
```shell-session
$ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync
```
2. Create the config.hcl file and configure the options for your use case. Refer to the [configuration reference](/docs/nia/configuration) for details about all CTS configurations.
3. Run Consul-Terraform-Sync (CTS).
```shell-session
$ consul-terraform-sync start -config-file <config.hcl>
```
4. Check status of tasks. Replace port number if configured in Step 2. Refer to [Consul-Terraform-Sync API](/docs/nia/api) for additional information.
```shell-session
$ curl localhost:8558/status/tasks
```
## Other Run modes
You can [configure CTS for high availability](/docs/nia/usage/run-ha), which is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected.
You can start CTS in [inspect mode](/docs/nia/cli/start#modes) to review and test your configuration before applying any changes. Inspect mode allows you to verify that the changes work as expected before running them in an unsupervised daemon mode.
For hands-on instructions on using inspect mode, refer to the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial.

25
website/data/docs-nav-data.json
View File

@ -794,16 +794,29 @@
"path": "nia/installation/install"
},
{
"title": "Requirements",
"path": "nia/installation/requirements"
},
{
"title": "Configure",
"title": "Configuration",
"path": "nia/installation/configure"
}
]
},
{
"title": "Usage",
"routes": [
{
"title": "Requirements",
"path": "nia/usage/requirements"
},
{
"title": "Run Consul-Terraform-Sync",
"path": "nia/installation/run"
"path": "nia/usage/run"
},
{
"title": "Run Consul-Terraform-Sync with High Availability",
"path": "nia/usage/run-ha"
},
{
"title": "Error Messages",
"path": "nia/usage/errors-ref"
}
]
},

BIN
website/public/img/nia/cts-ha-after.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

1
website/public/img/nia/cts-ha-after.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 67 KiB

BIN
website/public/img/nia/cts-ha-before.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 28 KiB

1
website/public/img/nia/cts-ha-before.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 68 KiB

20
website/redirects.js
View File

@ -1278,6 +1278,16 @@ module.exports = [
destination: '/docs/k8s/installation/vault/data-integration/connect-ca',
permanent: true,
},
{
source: '/docs/nia/installation/run',
destination: '/docs/nia/usage/run',
permanent: true,
},
{
source: '/docs/nia/installation/requirements',
destination: '/docs/nia/usage/requirements',
permanent: true,
},
{
source: '/docs/k8s/installation/install#consul-k8s-cli-installation',
destination: '/docs/k8s/installation/install-cli',
@ -1534,4 +1544,14 @@ module.exports = [
destination: '/docs/k8s/deployment-configurations/vault/wan-federation',
permanent: true,
},
{
source: '/docs/nia/installation/run',
destination: '/docs/nia/usage/run',
permanent: true,
},
{
source: '/docs/nia/installation/requirements',
destination: '/docs/nia/usage/requirements',
permanent: true,
},
]