diff --git a/website/content/docs/nia/api/index.mdx b/website/content/docs/nia/api/index.mdx index 3e38803027..07b1d396e6 100644 --- a/website/content/docs/nia/api/index.mdx +++ b/website/content/docs/nia/api/index.mdx @@ -2,32 +2,29 @@ layout: docs page_title: Consul-Terraform-Sync API description: >- - How to use the Consul-Terraform-Sync API + Consul-Terraform-Sync provides an API interface that lets you query CTS instance health, check CTS instance status, and run CTS tasks. --- -# Consul-Terraform-Sync API +# Consul-Terraform-Sync API Overview -When running in [daemon mode](/docs/nia/cli#daemon-mode), Consul-Terraform-Sync (CTS) serves an HTTP API interface. Details of the available API endpoints is available in the navigation to the left. +Consul-Terraform-Sync (CTS) provides an HTTP API interface for querying CTS instances and running and managing tasks. -### Port +## Port The API is served at the default port `8558` or a different port if set with [`port` configuration](/docs/nia/configuration#port) -### Version Prefix +## Version prefix All API routes are prefixed with `/v1/`. This documentation is for v1 of the API, which is the only version currently. Example: `localhost:8558/v1/status` -### Error +## Request ID -Successful API requests will receive a 2XX success status code. For other unsuccessful status codes, when possible, more details will be provided in a response body containing an error object. +Each call to a CTS API endpoint returns a `request_id` field. The field is a string generated by the API. Example: -Example: Status 400 Bad Request -```json -{ - "error": { - "message": "example error message: unsupported status parameter value" - } -} -``` +`"request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88"` + +## Error messages + +The API sends a response code in the 200 range if the call is successful. If the call is unsuccessful, the API sends an error message that includes additional information when possible. Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information. \ No newline at end of file diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index edfd512072..108f2c9aa0 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -6,28 +6,27 @@ description: >- --- # Status -The `/status` endpoints share status-related information for tasks. This information is available for understanding the status of individual tasks and across tasks. +The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). -The health status value is determined by aggregating the success or failure of the event of a task detecting changes in Consul services and then updating network infrastructure. Currently, only the 5 most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). +If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send requests to the [`/status/cluster` endpoint path](#cluster-status) on any CTS cluster member instance to receive information about the entire cluster. Calling the `status` endpoint path (without `/cluster`), however, returns a 400 error if the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. -## Overall Status +## Status for all tasks -This endpoint currently returns the overall status information for all tasks. +This endpoint returns the overall status information for all tasks. | Method | Path | Produces | | ------ | ------------------- | ------------------ | | `GET` | `/status` | `application/json` | -### Request Parameters -Currently no request parameters are offered for the overall status API. +### Request parameters -### Response Statuses +No request parameters are available for this endpoint. -| Status | Reason | -| ------ | ---------------- | -| 200 | Successfully retrieved the status | +### Response statuses -### Response Fields +- `200`: Successfully retrieved the status. + +### Response fields | Name | Type | Description | | ----- | ------ | ------------------ | @@ -65,7 +64,7 @@ Response: } ``` -## Task Status +## Task status This endpoint returns the individual task status information for a single specified task or for all tasks. @@ -79,7 +78,7 @@ Task health status value is determined by the success or failure of all stored [ | ------ | ------------------- | ------------------ | | `GET` | `/status/tasks/:task_name` | `application/json` | -### Request Parameters +### Request parameters | Name | Required | Type | Description | Default | | -------- | -------- | ------ | ------------------ | ------- | @@ -87,14 +86,12 @@ Task health status value is determined by the success or failure of all stored [ |`include` | Optional | string | Only accepts the value `"events"`. Use to include stored event information in response. | none |`status` | Optional | string | Only accepts health status values `"successful"`, `"errored"`, `"critical"`, or `"unknown"`. Use to filter response by tasks that have the specified health status value. Recommend setting this parameter when requesting all tasks i.e. no `task` parameter is set. | none -### Response Statuses +### Response statuses -| Status | Reason | -| ------ | ---------------- | -| 200 | Successfully retrieved the task status | -| 404 | Task with the given name not found | +- `200`: Successfully retrieved the task status +- `404`: Task with the given name not found -### Response Fields +### Response fields The response is a JSON map of task name to a status information structure with the following fields. @@ -126,7 +123,11 @@ Event represents the process of updating network infrastructure of a task. The d |`config.source` | string | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** Module configured for the task. |`config.providers` | list[string] | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** List of the providers configured for the task. -### Example: All Task Statuses +### Examples + +The following examples call `status` endpoints to retrieve information about tasks. + +#### All task statuses Request: ```shell-session @@ -163,7 +164,7 @@ Response: } ``` -### Example: Individual Task Status with Events +#### Individual task status with events Request: ```shell-session @@ -225,3 +226,78 @@ Response: } } ``` + +## Cluster status + +The `/v1/status/cluster` API endpoint returns information about high-availability clusters and its members, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. + +| Method | Path | Response format | +| ------ | ----------------- | ------------------ | +| `GET` | `/status/cluster` | `application/json` | + +### Request parameters + +Currently no request parameters are offered for the cluster status API. + +### Request statuses + +- `200`: Successfully retrieved the cluster status + +### Response fields + +The following table describes the responses that the `status/cluster` endpoint can send. + +| Field | Type | Description | +| --- | ---- | --- | +| `cluster_name` | string | Identifies the name of the cluster. | +| `members` | list[[member](#member-objects)] | Contains a list of [member objects](#member-objects). Each object in the `members` list represents a CTS instance. | + +#### Member objects + +The following table describes the fields available for objects in the `members` array. + +| Field | Type | Description | +| --- | ---- | --- | +| `address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured on the leader instance. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | +| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | +| `id` | string | Indicates the service registration ID. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | +| `leader` | boolean | Identifies the cluster leader. | +| `service_name` | string | Identifies the name of the service that the instance represents. The value is set by the `service_name` field in the [Service Registration](/docs/nia/configuration#service-registration) configuration. | + +### Example + +The following command calls the `/status/cluster` endpoint: + +```shell-session +$ curl –request GET http://localhost:8553/v1/status/cluster +``` + +The following example response shows three CTS instances. The cluster leader is `cts-02` and is advertising that its address is `cts-02.example.com`: + +```json +{ + "cluster_name": "cluster-a", + "members": [ + { + "address": "cts-02.example.com", + "healthy": true, + "id": "cts-02", + "leader": true, + "service_name": "consul-terraform-sync" + }, + { + "healthy": true, + "id": "cts-03", + "leader": false, + "service_name": "consul-terraform-sync" + }, + { + "healthy": true, + "id": "cts-01", + "leader": false, + "service_name": "consul-terraform-sync" + } + ], + "request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88" +} +``` \ No newline at end of file diff --git a/website/content/docs/nia/api/tasks.mdx b/website/content/docs/nia/api/tasks.mdx index b870d26718..60ca787a3b 100644 --- a/website/content/docs/nia/api/tasks.mdx +++ b/website/content/docs/nia/api/tasks.mdx @@ -9,6 +9,8 @@ description: >- The `/tasks` endpoints interact with the tasks that Consul-Terraform-Sync (CTS) is responsible for running. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. + ## Get Tasks This endpoint returns information about all existing tasks. diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index a6712a9733..39ba71f654 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -2,37 +2,29 @@ layout: docs page_title: Architecture description: >- - Consul-Terraform-Sync Architecture + Learn about the Consul-Terraform-Sync architecture and high-level CTS components, such as the Terraform driver and tasks. --- # Consul-Terraform-Sync Architecture -Consul-Terraform-Sync (CTS) is a service-oriented tool for managing -network infrastructure near real-time. CTS runs as a daemon -and integrates the network topology maintained by your Consul cluster with your -network infrastructure to dynamically secure and connect services. +Consul-Terraform-Sync (CTS) is a service-oriented tool for managing network infrastructure near real-time. CTS runs as a daemon and integrates the network topology maintained by your Consul cluster with your network infrastructure to dynamically secure and connect services. -## 10,000 Foot View +## CTS workflow + +The following diagram shows the CTS workflow as it monitors the Consul service catalog for updates. [![Consul-Terraform-Sync Architecture](/img/nia-highlevel-diagram.svg)](/img/nia-highlevel-diagram.svg) -The diagram shows CTS monitoring the Consul service catalog -for updates and utilizing Terraform to update the state of the infrastructure. +1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and views](#watcher-and-views). +1. CTS detects a change. +1. CTS prompts Terraform to update the state of the infrastructure. -There are two principal aspects of Sync to know about corresponding to the -lines to Consul and Terraform in the diagram above. The line to Consul -represents the Watchers monitoring the state of Consul's service catalog (and -possibly KV store) while the line to Terraform represents tasks being run to -update the infrastructure. -## Watcher and Views +## Watcher and views -CTS monitors Consul for updates utilizing Consul's [Blocking -Queries](/api-docs/features/blocking) whenever supported, falling back on -polling when not. The watcher maintains a separate thread (known internally as -a view) for each value monitored, running any tasks that depend on that watched -value whenever it's updated. Say, for example, running a task to update a -proxy when an instance goes unhealthy. +CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as *watchers*. + +The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it is updated. These threads are referred to as _views_. For example, a thread may run a task to update a proxy when the watcher detects that an instance has become unhealthy . ## Tasks @@ -53,8 +45,35 @@ network infrastructure. The following [drivers](/docs/nia/network-drivers#terraf Each driver includes a set of providers that [enables support](/docs/nia/terraform-modules) for a wide variety of infrastructure applications. -## Security Guidelines +## State storage and persistence -The [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) -tutorial contains a checklist of best practices to secure your -CTS installation for a production environment. +The following types of state information are associated with CTS. + +### Terraform state information + +By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). The data persists if CTS stops and the backend is configured to a remote location. + +### CTS task and event data + +By default, CTS stores task and event data in memory. This data is transient and does not persist. If you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha), CTS stores the data in the Consul KV. High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. + +The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). + +## Instance compatibility checks (high availability) + +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage. + +The CTS instance compatibility check reports an error if the task [module](/docs/nia/configuration#module) is configured with a local module, but the module does not exist on the CTS instance. Refer to the [Terraform documentation](https://www.terraform.io/language/modules/sources#module-sources) for additional information about module sources. Example log: + +```shell-session +[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" +``` +Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information. + +CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. + +CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This can happen when all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. + +## Security guidelines + +We recommend following the network security guidelines described in the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. The tutorial contains a checklist of best practices to secure your CTS installation for a production environment. diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 04534d4a38..c47a183df6 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -2,61 +2,42 @@ layout: docs page_title: Start Command description: >- - Consul-Terraform-Sync supports start command for starting CTS as a daemon + Run the Consul-Terraform-Sync start command to start the CTS daemon. --- # start -The `start` command starts Consul-Terraform-Sync (CTS) as a daemon. When CTS runs as a daemon, there is no default configuration to start CTS. You must set a configuration flag `-config-file` or `-config-dir`. For example: +The `start` command starts the Consul-Terraform-Sync (CTS) daemon. + +## Usage ```shell-session -$ consul-terraform-sync start -config-file=config.hcl +$ consul-terraform-sync start -config-file [OPTIONS] ``` -To review a list of available flags, use the `-help` or `-h` flag. - -## Modes - -CTS can be run as a daemon in different modes. - -### Long-running Mode - -Flag: none - -Behavior: This is the default mode in which CTS passes through a once-mode phase and then turns into a long-running process. During the once-mode phase, the daemon will exit with a non-zero status if it encounters an error. After successfully passing through once-mode phase, it will begin a long-running process in which errors are logged and exiting is not expected behavior. When the long-running process begins, CTS daemon starts serving API and command requests. - -### Inspect Mode - -Flag: `-inspect` - -Behavior: CTS will display the proposed state changes for all tasks once and exit. No changes are applied in this mode. On encountering an error before completing, CTS will exit with a non-zero status. - -Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks would update network infrastructure as expected. - ---- - -Flag: `-inspect-task [task-name]` - -Behavior: This has similar behavior as `-inspect` mode for the selected task. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. - -Usage: Useful to debug one or more tasks to confirm configuration is accurate and the selected tasks would update network infrastructure as expected. - -### Once Mode - -Flag: `-once` - -Behavior: CTS will run all tasks once with buffer periods disabled and exit. On encountering an error before completing, CTS will exit with a non-zero status. - -Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks update network infrastructure as expected. +The `-config-file` or `-config-dir` flag is required. Use the flag to specify the [CTS instance configuration file](/docs/nia/configuration) or directory containing several configuration files to start a CTS cluster. ## Options -The `start` command supports the following options: +The following table describes all of the available flags. -| Name | Required | Type | Description |Default | -| --------------- | -------- | ------- | ------------------------------- | ----------------------- | -| `-config-dir `      | Required when`-config-file` is not set | string | A directory to load files for configuring CTS. Configuration files require an .hcl or .json file extension in order to specify their format. This option can be specified multiple times to load different directories. | none | -| `-config-file`                        | Required when `-config-dir` is not set | string | A file to load for configuring CTS. Configuration file requires an .hcl or .json extension in order to specify their format. This option can be specified multiple times to load different configuration files. | none | -| `-inspect` | Optional | boolean | Run CTS in Inspect mode to print the proposed state changes for all tasks, and then exit. No changes are applied in this mode. | false | -| `-inspect-task`                             | Optional | string | Run CTS in Inspect mode to print the proposed state changes for the task, and then exit. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. | none | -| `-once` | Optional | boolean | Render templates and run tasks once. Does not start the process in long-running mode and disables buffer periods. | false | +| Name | Required | Type | Description |Default | +| --- | ---- | ---- | ---- | ---- | +| `-config-dir`| Required when `-config-file` is not set | string | Specifies a directory containing CTS instance configuration files to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one directory of files. | none | +| `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | +| `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | +| `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | +| `-once` | Optional | boolean | Starts CTS in once-mode. In once-mode, CTS renders templates, runs tasks once, and disables buffer periods. Refer to [Modes](#modes) for additional information. | `false` | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | `false` | +| `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | + +## Modes + +By default, CTS starts in long-running mode. The following table describes all available CTS modes. + +| Name | Description |How to start | +| --- | --- | --- | +| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process.

When the long-running process begins, the CTS daemon serves API and command requests.

If an error occurs, CTS logs it and continues running.

| No additional flags.
This is the default mode. | +| Once-mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | +| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks will update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| +| High availability mode | A long-running process that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/compatibility.mdx b/website/content/docs/nia/compatibility.mdx index c4cc632858..05e301e028 100644 --- a/website/content/docs/nia/compatibility.mdx +++ b/website/content/docs/nia/compatibility.mdx @@ -7,7 +7,7 @@ description: >- # Compatibility -The following tables list the Consul-Terraform-Sync (CTS) version compatibility for Consul, Terraform, and Vault. +This topic describes Consul-Terraform-Sync (CTS) cross-compatibility with Consul, Terraform, and Vault. ## Consul @@ -26,8 +26,10 @@ CTS integration with Terraform is supported for the following: | CTS Version | Terraform CLI Version | Terraform Cloud Version | Terraform Enterprise Version | | :------------------ | :-------------------- | :---------------------- | :--------------------------- | +| CTS Enterprise 0.7 | 0.13-1.2 | Latest | v202010-2 - Latest | | CTS Enterprise 0.4+ | 0.13 - 1.1 | Latest | v202010-2 - Latest | | CTS Enterprise 0.3 | 0.13 - 1.1 | N/A | v202010-2 - Latest | +| CTS OSS 0.7 | 0.13 - 1.2 | N/A | N/A | | CTS OSS 0.3+ | 0.13 - 1.1 | N/A | N/A | | CTS OSS 0.2 | 0.13 - 1.0 | N/A | N/A | | CTS OSS 0.1 | 0.13 - 0.14 | N/A | N/A | diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index c8ae16e123..36bff7f2dc 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -5,11 +5,11 @@ description: >- Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the consul-terraform-sync daemon. --- -# Configuration Options for Consul-Terraform-Sync +# Consul-Terraform-Sync Configuration -The Consul-Terraform-Sync (CTS) daemon is configured using configuration files and supports [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) (HCL) and JSON file formats. +This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL or JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. -## Global Config Options +## Global configurations Top level options are reserved for configuring CTS. @@ -44,7 +44,7 @@ tls { - `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - `log_level` - (string: "INFO") The log level to use for CTS logging. This defaults to "INFO". The available log levels are "TRACE", "DEBUG", "INFO", "WARN", and "ERR". - `port` - (int: 8558) The port for CTS to use to serve API requests. -- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. Used as the service ID for CTS if service registration is enabled. +- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in a high availability cluster. - `syslog` - Specifies the syslog server for logging. - `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging. - `facility` - (string: "local0") Name of the syslog facility to log to. @@ -82,7 +82,7 @@ license { | `auto_retrieval` | Optional | object | Configures the license auto-retrieval used by CTS. To learn more, review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for details | Review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for defaults. | -#### Auto-Retrieval +### Auto-retrieval You can use the `auto_retrieval` block to configure the the automatic license retrieval in CTS. When enabled, CTS attempts to retrieve a new license from its configured Consul Enterprise backend once a day. If CTS cannot retrieve a license and the current license is reaching its expiration date, CTS attempts to retrieve a license with increased frequency, as defined by the [License Expiration Date Handling](/docs/nia/enterprise/license#license-expiration-handling). @@ -92,7 +92,7 @@ You can use the `auto_retrieval` block to configure the the automatic license re | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | string | If set to true, enables license auto-retrieval | true | -## Consul +## Consul connection The `consul` block configures the CTS connection with a Consul agent so that CTS can perform queries for task execution. It also configures the automatic registration of CTS as a service with Consul. @@ -127,7 +127,7 @@ consul { | `transport` | Optional | [transport](/docs/nia/configuration#transport) | Low-level network connection details || | `service_registration` | Optional| [service_registration](/docs/nia/configuration#service-registration) | Options for how CTS will register itself as a service with a health check to Consul. || -##### ACL Requirements +### ACL requirements The following table describes the ACL policies required by CTS. For more information, refer to the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync) tutorial. | Policy | Resources | @@ -140,7 +140,7 @@ The following table describes the ACL policies required by CTS. For more informa | `keys:write` | `consul-terraform-sync/` Only required when using Consul as the [Terraform backend](/docs/nia/configuration#backend). | -#### Auth +### Auth Configures HTTP basic authentication for communicating with Consul. | Parameter | Required | Type | Description | Default | @@ -149,7 +149,8 @@ Configures HTTP basic authentication for communicating with Consul. | `username` | Optional | string | Username for authentication| none | | `password` | Optional | string | Password for authentication| none | -#### TLS +### TLS + Configure TLS to use a secure client connection with Consul. Using HTTP/2 can solve issues related to hitting Consul's maximum connection limits, as well as improve efficiency when processing many blocking queries. This option is required for Consul-Terraform-Sync when connecting to a [Consul agent with TLS verification enabled for HTTPS connections](/docs/agent/config/config-files#verify_incoming). If Consul is using a self-signed certificate that you have not added to the global CA chain, you can set this certificate with `ca_cert` or `ca_path`. Alternatively, you can disable SSL verification by setting `verify` to false. However, disabling verification is a potential security vulnerability. @@ -164,7 +165,7 @@ If Consul is using a self-signed certificate that you have not added to the glob | `key` | Optional | string | The path to the PEM-encoded private key file used with the client certificate configured by `cert`. Required if Consul has `verify_incoming` set to true.

Can also be provided through the `CONSUL_CLIENT_KEY` environment variable. | none | | `server_name` | Optional | string | The server name to use as the Server Name Indication (SNI) for Consul when connecting via TLS.

Can also be provided through the `CONSUL_TLS_SERVER_NAME` environment variable. | none | -#### Transport +### Transport Configures the low-level network connection details to Consul. To achieve the shortest latency between a Consul service update to a task execution, configure `max_idle_conns_per_host` equal to or greater than the number of services in automation across all tasks. This value should be lower than the configured [`http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the Consul agent. @@ -182,15 +183,15 @@ If `max_idle_conns_per_host` and the number of services in automation is greater | `tls_handshake_timeout` | Optional | string | The amount of time to wait to complete the TLS handshake. | `10s`| -#### Service Registration +### Service registration CTS automatically registers itself with Consul as a service with a health check, using the [`id`](/docs/nia/configuration#id) configuration as the service ID. CTS deregisters itself with Consul when CTS stops gracefully. If CTS is unable to register with Consul, then it will log the error and continue without exiting. Service registration requires that the [Consul token](/docs/nia/configuration#consul) has an ACL policy of `service:write` for the CTS service. | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | -| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul.| `true` | -| `service_name` | Optional | string | The service name for CTS. | `Consul-Terraform-Sync` | +| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. | `true` | +| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high-availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | @@ -213,9 +214,38 @@ The default health check is an [HTTP check](/docs/discovery/checks#http-interval | Timeout | `2s` | | TLSSkipVerify | `false` | +## High availability + +Add a `high_availability` block to your configuration to enable CTS to run in high availability mode. Refer to [Run Consul-Terrform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. The `high_availability` block contains the following configuration items. + +### High availability cluster + +The `cluster` parameter contains configurations for the cluster you want to operate with high availability enabled. You can configure the following options: + +| Parameter | Description| Required | Type | +| --------- | ---------- | -------- | ------| +| `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `"consul"` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | + +#### High availability cluster storage + +The `high_availability.cluster.storage` object contains the following configurations. + +| Parameter | Description| Required | Type | +| --------- | ---------- | -------- | ------| +| `parent_path` | Defines a parent path in the Consul KV for CTS to store state information. Default is `consul-terraform-sync/`. CTS automatically appends the cluster name to the parent path, so the effective default directory for state information is `consul-terraform-sync/`. | Optional | String | +| `namespace` | Specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | +| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | String | + +### High availability instance + +The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: +- `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. + + ## Service -~> Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. See [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. +~> **Note:** Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. Refer to [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. A `service` block is an optional block to explicitly define the services configured in the `task` block's `services` field (deprecated). `service` blocks do not define services configured in the `task` block's `condition "services"` or `module_input "services` blocks. @@ -241,7 +271,7 @@ service { ## Task -A `task` block configures which task to execute in automation. When the task should execute can be determined by the `service` block (deprecated) and `condition` block. The `task` block may be specified multiple times to configure multiple tasks. +A `task` block configures which task to execute in automation. Use the `condition` block to specify when the task executes. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). ```hcl task { @@ -624,6 +654,7 @@ driver "terraform" { - Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html). Visit the Terraform documentation links for details on backend configuration options. - If omitted, CTS will generate default values and use configurations from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html), which stores Terraform statefiles in the Consul KV. The [ACL token provided for Consul authentication](#consul) is used to read and write to the KV store and requires [Consul KV privileges](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifier appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. - The remote enhanced backend is not supported with the Terraform driver to run operations in Terraform Cloud. Use the [Terraform Cloud driver](#terraform-cloud-driver) to integrate CTS with Terraform Cloud for remote workspaces and remote operations. + - The `local` backend type is not supported with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. - `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the CTS log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. - `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the CTS daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path. - `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. diff --git a/website/content/docs/nia/installation/requirements.mdx b/website/content/docs/nia/installation/requirements.mdx deleted file mode 100644 index 17f13ff5eb..0000000000 --- a/website/content/docs/nia/installation/requirements.mdx +++ /dev/null @@ -1,129 +0,0 @@ ---- -layout: docs -page_title: Requirements -description: >- - Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon. ---- - -# Prerequisites Needed to Run Consul-Terraform-Sync - -The following components are required to run Consul-Terraform-Sync (CTS): - -* A Terraform Provider -* A Terraform Module -* A Consul cluster running outside of the `consul-terraform-sync` daemon - -Practitioners can add support for their network infrastructure through Terraform providers. Once network infrastructure support exists, practitioners can add network integrations in the form of Terraform modules. - -The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). - -## Run a Consul Cluster - -Below are several steps towards a minimum Consul setup required for running CTS. - -### Install Consul - -CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately. - -To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install). - -For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul). - -### Run an Agent - -The Consul agent must be running in order to dynamically update network devices. To run the local Consul agent, you can run Consul in development mode which can be started with `consul agent -dev` for simplicity. For more details on running Consul agent, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). - -When running a Consul agent with CTS in production, we suggest to keep a few considerations in mind. CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, like changes to registered services. This results in multiple long running TCP connections between CTS and the agent to poll changes for each dependency. Monitoring a high number of services may quickly hit the default Consul agent connection limits. - -There are 2 ways to fix this issue. The first and recommended fix is to use HTTP/2 (requires HTTPS) to communicate between Consul-Terraform-Sync and the Consul agent. When using HTTP/2 only a single connection is made and reused for all communications. See the [Consul Configuration section](/docs/nia/configuration#consul) for more. The other option is to configure [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the agent to a reasonable value proportional to the number of services monitored by Consul-Terraform-Sync. - -### Register Services - -CTS monitors Consul catalog for service changes which lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent either by loading a service definition or by HTTP API request. - -If you are running Consul in development mode, below is an example of registering a service by HTTP API request: - -```shell-session -$ echo '{ - "ID": "web", - "Name": "web", - "Address": "10.10.10.10", - "Port": 8000 -}' > payload.json - -$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register -``` - -The above example registers a service named "web" with your Consul agent. This represents a non-existent web service running at 10.10.10.10:8000. Your web service is now available for CTS to consume. You can have CTS monitor the web service to execute a task and update network device(s) by configuring "web" in [`condition "services"`](/docs/nia/configuration#services-condition) of a task block. If the web service has any non-default values, it can also be configured in `condition "services"`. - -For more details on registering a service by HTTP API request, refer to the [register service API docs](/api-docs/agent/service#register-service). - -For more details on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started). - -### Run a Cluster - -The previous steps of installing and running a single Consul agent then registering a single service is sufficient to meaningfully start running CTS. - -If you would like to run a Consul cluster rather than a single agent, refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started). This will walk you through the steps of running multiple Consul agents and then joining them together into a cluster. - -## Network Infrastructure (using a Terraform Provider) - -CTS integrations for the Terraform driver utilizes Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver of CTS inherits the expansive collection of Terraform providers to integrate with, and with release of [Terraform 0.13](https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-13/), this extends to include providers written by the community too by using [provider source](https://www.hashicorp.com/blog/adding-provider-source-to-hashicorp-terraform/). - -### Finding Terraform Providers - -To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers). - -### How to Create a Provider - -If there is no existing Terraform provider, a new Terraform provider can be [created](https://learn.hashicorp.com/tutorials/terraform/provider-setup) and [published](https://www.terraform.io/docs/registry/providers/publishing.html). The provider can then be used within a network integration task by authoring a compatible Terraform module. - -## Network Integration (using a Terraform Module) - -The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module along with how it is configured within a task determines the condition under which your infrastructure is updated. - -Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. - -Continue to the next page to [get started with configuring CTS and how to use Terraform providers and modules for tasks.](/docs/nia/installation/configure) - -### Partner Terraform Modules - -The modules listed below are available to use and are compatible with CTS. - -#### A10 Networks - -- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia) - -#### Avi Networks - -- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool) - -#### AWS Application Load Balancer (ALB) - -- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia) - -#### Checkpoint - -- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia) - -#### Cisco ACI - -- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia) -- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia) - -#### Citrix ADC - -- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) - -#### F5 - -- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia) - -#### NS1 - -- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) - -#### Palo Alto Networks - -- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia) -- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia) diff --git a/website/content/docs/nia/installation/run.mdx b/website/content/docs/nia/installation/run.mdx deleted file mode 100644 index d4746eb03d..0000000000 --- a/website/content/docs/nia/installation/run.mdx +++ /dev/null @@ -1,34 +0,0 @@ ---- -layout: docs -page_title: Run Consul-Terraform-Sync -description: >- - Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the `consul-terraform-sync` daemon. ---- - -# Run Consul-Terraform-Sync - -1. Move the `consul-terraform-sync` binary to a location available on your `PATH`. - - ```shell-session - $ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync - ``` - -2. Create the config.hcl file, all the options are available [here](/docs/nia/configuration). - -3. Run Consul-Terraform-Sync (CTS). - - ```shell-session - $ consul-terraform-sync start -config-file - ``` - -4. Check status of tasks. Replace port number if configured in Step 2. See additional API endpoints [here](/docs/nia/api) - - ```shell-session - $ curl localhost:8558/status/tasks - ``` - -## Other Run modes - -CTS allows you to inspect your configuration before applying any change and to run in once mode, meaning that you can verify the changes are correctly applied in a test run before running it in unsupervised daemon mode. - -To learn more on these options check the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx new file mode 100644 index 0000000000..0b24a1189a --- /dev/null +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -0,0 +1,143 @@ +--- +layout: docs +page_title: Error Messages +description: >- + Look up Consul-Terraform-Sync error message to learn how to resolve potential issues using CTS. +--- + +# Error Messages + +This topic explains error messages you may encounter when using Consul-Terraform-Sync (CTS). + +## Example error log messages + +If you configured the CTS cluster to run in [high availability mode](/docs/nia/usage/run-ha) and the the local module is missing, then the following message appears in the log: + +```shell-session +[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" +``` + +The resolution is to add the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. + +## Example API and CLI error messages + +**Error**: + +```json +{ + "error": { + "message": "redirect requests to leader 'cts-01' at cts-01.example.com:8558" + } +} +``` + +**Conditions**: + +- CTS can determine the leader. +- `high_availability.instance.address` is configured for the leader. +- The CTS instance you sent the request to is not the leader. + + + +**Resolution**: + +Redirect the request to the leader instance, for example: + +```shell-session +$ curl --request GET cts-01.example.com:8558/v1/tasks +``` +--- + + +**Error**: + +```json +{ + "error": { + "message": "redirect requests to leader 'cts-01'" + } +} +``` + +**Conditions**: + +* CTS can determine the leader. +* The CTS instance you sent the request to is not the leader. +* `high_availability.instance.address` is not configured. + +**Resolution**: + +Identify the leader instance address and redirect the request to the leader. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: + +```shell-session +[INFO] ha: acquired leadership lock: id=. + +We recommend deploying a cluster that has three instances. + +--- + +**Error**: + +```json +{ + "error": { + "message": "redirect requests to leader" + } +} +``` + +**Conditions**: + +* The CTS instance you sent the request to is not the leader. +* The CTS is unable to determine the leader. +* Note that these conditions are rare. + + +**Resolution**: + +Identify and send the request to the leader CTS instance. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: + +```shell-session +[INFO] ha: acquired leadership lock: id= +``` + +--- + +**Error**: + +```json +{ + "error": { + "message": "this endpoint is only available with high availability configured" + } +} +``` + +**Conditions**: + +- You called the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) without configuring CTS for [high availability](/docs/nia/usage/run-ha). + +**Resolution**: + +Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). + +--- + +**Error**: + +```json +{ + "error": { + "message": "example error message: unsupported status parameter value" + } +} +``` + +**Conditions**: + +- You sent a request to the `status` API endpoint. +- The request included an unsupported parameter value. + +**Resolution**: + +Send a new request and verify that all of the parameter values are correct. \ No newline at end of file diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx new file mode 100644 index 0000000000..eeb6796649 --- /dev/null +++ b/website/content/docs/nia/usage/requirements.mdx @@ -0,0 +1,134 @@ +--- +layout: docs +page_title: Requirements +description: >- + Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon. +--- + +# Requirements + +The following components are required to run Consul-Terraform-Sync (CTS): + +- A Terraform provider +- A Terraform module +- A Consul cluster running outside of the `consul-terraform-sync` daemon + +You can add support for your network infrastructure through Terraform providers so that you can apply Terraform modules to implement network integrations. + +The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driver has [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). + +## Run a Consul cluster + +Below are several steps towards a minimum Consul setup required for running CTS. + +### Install Consul + +CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately. + +To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install). + +For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul). + +### Run an agent + +The Consul agent must be running in order to dynamically update network devices. Refer to the [Consul agent documentation](/docs/agent/index) for information about configuring and starting a Consul agent. For hands-on instructions about running Consul agents, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). + +When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitoring a high number of services. + +To avoid reaching the limit prematurely, we recommend using HTTP/2 (requires HTTPS) to communicate between CTS and the Consul agent. When using HTTP/2, CTS establishes a single connection and reuses it for all communication. Refer to the [Consul Configuration section](/docs/nia/configuration#consul) for details. + +Alternatively, you can configure the [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) option to set a maximimum number of connections to meet your needs. + +### Register services + +CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon is operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. + +The following HTTP API request example registers a service named `web` with your Consul agent: + +```shell-session +$ echo '{ + "ID": "web", + "Name": "web", + "Address": "10.10.10.10", + "Port": 8000 +}' > payload.json + +$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register +``` + +The example represents a non-existent web service running at `10.10.10.10:8000` that is now available for CTS to consume. + +You can configure CTS to monitor the web service, execute a task, and update network device(s) by configuring `web` in the [`condition "services"`](/docs/nia/configuration#services-condition) task block. If the web service has any non-default values, it can also be configured in `condition "services"`. + +For more details on registering a service using the HTTP API endpoint, refer to the [register service API docs](/api-docs/agent/service#register-service). + +For hands-on instructions on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started). + +### Run a cluster + +For production environments, we recommend operating a Consul cluster rather than a single agent. Refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started) for instructions on starting multiple Consul agents and joining them into a cluster. + +## Network infrastructure using a Terraform provider + +CTS integrations for the Terraform driver use Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver for CTS inherits the expansive collection of Terraform providers to integrate with. You can also specify a provider `source` in the [`required_providers` configuration](https://www.terraform.io/language/providers/requirements#requiring-providers) to use providers written by the community (requires Terraform 0.13 or later). + +### Finding Terraform providers + +To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers). + +### How to create a provider + +If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registry so that you can use it within a network integration task or create a compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: + +- [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) +- [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). + +## Network integration using a Terraform module + +The Terraform module for a task in CTS is the core component of the integration. It declares which resources to use and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the conditions under which your infrastructure is updated. + +Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver. You can also use a [module built by partners](#partner-terraform-modules). + +Refer to [Configuration](/docs/nia/configuration) for information about configuring CTS and how to use Terraform providers and modules for tasks. + +### Partner Terraform Modules + +The modules listed below are available to use and are compatible with CTS. + +#### A10 Networks + +- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia) + +#### Avi Networks + +- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool) + +#### AWS Application Load Balancer (ALB) + +- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia) + +#### Checkpoint + +- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia) + +#### Cisco ACI + +- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia) +- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia) + +#### Citrix ADC + +- Create, Update, and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) + +#### F5 + +- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia) + +#### NS1 + +- Create, Delete, and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) + +#### Palo Alto Networks + +- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia) +- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia) diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx new file mode 100644 index 0000000000..81fa5c2d16 --- /dev/null +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -0,0 +1,181 @@ +--- +layout: docs +page_title: Run Consul-Terraform-Sync with High Availability +description: >- + Improve network automation resiliency by enabling high availability for Consul-Terraform-Sync. HA enables persistent task and event data so that CTS functions as expected during a failover event. +--- + +# Run Consul-Terraform-Sync with High Availability + + Licenses are only required for Consul-Terraform-Sync (CTS) Enterprise + +This topic describes how to run Consul-Terraform-Sync (CTS) configured for high availability. High availability is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. + +## Introduction + +A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process when it is configured for high availability: + +1. The CTS cluster promotes a new leader from the pool of followers in the network. +1. The new leader begins running all existing tasks in `once-mode` in order to process changes that occurred during the failover transition period. In this mode, CTS runs all existing tasks one time. +1. The new leader logs any errors that occur during `once-mode` operation and the new leader continues to monitor Consul for changes. + +In a standard configuration, CTS exits if errors occur when the CTS instance runs tasks in `once-mode`. In a high availability configuration, CTS logs the errors and continues to operate without interruption. + +The following diagram shows operating state when high availability is enabled. CTS Instance A is the current leader and is responsible for monitoring and running tasks: + +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.svg) + +The following diagram shows the CTS cluster state after the leader stops. CTS Instance B becomes the leader responsible for monitoring and running tasks. + +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.svg) + +### Failover details + +- The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. +- If failover occurs during task execution, a new leader is elected. The new leader will attempt to run all tasks once before continuing to monitor for changes. +- If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. +- If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. +- If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. + +Note that driver behavior is consistent whether or not CTS is running in high availability mode. + +## Requirements + +Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. + +* CTS Enterprise 0.7 or later +* Terraform CLI 0.13 or later +* All instances in a cluster must be in the same datacenter. + +You must configure appropriate ACL permissions for your cluster. Refer to [ACL permissions](#) for details. + +We recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. + +## Configuration + +Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](/docs/nia/configuration#high-availability) for details about the configuration fields for the `high_availability` block. + +The following example configures high availability functionality for a cluster named `cts-cluster`: + + + +```hcl +high_availability { + cluster { + name = "cts-cluster" + storage "consul" { + parent_path = "cts" + namespace = "ns" + session_ttl = "30s" + } + } + + instance { + address = "cts-01.example.com" + } +} +``` + + +### ACL permissions + +The `session` and `keys` resources in your Consul environment must have `write` permissions. Refer to the [ACL documentation](/docs/security/acl) for details on how to define ACL policies. + +If the `high_availability.cluster.storage.namespace` field is configured, then your ACL policy must also enable `write` permissions for the `namespace` resource. + +## Start a new CTS cluster + +We recommend deploying a cluster that includes three CTS instances. This is so that the cluster has one leader and two followers. + +1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/configuration) for all configuration options. +1. Issue the startup command and pass the configuration file. Refer to the [`start` command reference](/docs/nia/cli/start#modes) for additional information about CTS startup modes. + ```shell-session + $ consul-terraform-sync start -config-file ha-config.hcl + ``` +1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Only the leader of the cluster will return a successful response. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. + + ```shell-session + $ curl localhost:/status/tasks + ``` + +Repeat the procedure to start the remaining instances for your cluster. We recommend using near-identical configurations for all instances in your cluster. You may not be able to use exact configurations in all cases, but starting instances with the same configuration improves consistency and reduces confusion if you need to troubleshoot errors. + +## Modify an instance configuration + +You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). + +1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: + ```shell-session + [INFO] ha: acquired leadership lock: id= + ``` +1. Stop one of the follower CTS instances and apply the new configuration. +1. Restart the follower instance. +1. Repeat steps 2 and 3 for other follower instances in your cluster. +1. Stop the leader instance. One of the follower instances becomes the leader. +1. Apply the new configuration to the former leader instance and restart it. + +## Modify tasks + +When high availability is enabled, CTS persists task and event data. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. + +You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes because inconsistencies between the state and the configuration can occur when mixing methods. + +### Delete and recreate the task (recommended) + +Use the CTS API to identify the CTS leader instance as well as delete and replace a task. + +1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: + + ```shell-session + [INFO] ha: acquired leadership lock: id= + ``` +1. Send a `DELETE` call to the [`/task/` endpoint](/docs/nia/api/tasks#delete-task) to delete the task. In the following example, the leader instance is at `localhost:8558`: + + ```shell-session + $ curl --request DELETE localhost:8558/v1/tasks/task_a + ``` + + You can also use the [`task delete` command](/docs/nia/cli/task#task-delete) to complete this step. + +1. Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. + ```shell-session + $curl --header "Content-Type: application/json" \ + --request POST \ + --data @payload.json \ + localhost:8558/v1/tasks +``` + + You can also use the [`task-create` command](/docs/nia/cli/task#task-create) to complete this step. + +### Discard data with the `-reset-storage` flag + +You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/start#options) to discard persisted data if you need to update a task. + +1. Stop a follower instance. +1. Update the instance’s task configuration. +1. Restart the instance and include the `-reset-storage` flag. +1. Stop all other instances so that the updated instance becomes the leader. +1. Start all other instances again. +1. Restart the instance you restarted in step 3 without the `-reset-storage` flag so that it starts up with the current state. If you continue to run an instance with the `-reset-storage` flag enabled, then CTS will reset the state data whenever the instance becomes the leader. + +## Troubleshooting + +Use the following troubleshooting procedure if a previous leader had been running a task successfully but the new leader logs an error after a failover: + +1. Check the logs printed to the console for errors. Refer to the [`syslog` configuration](/docs/nia/configuration#syslog) for information on how to locate the logs. In the following example output, CTS reported a `401: Bad credentials` error: + ```shell-session + 2022-08-23T09:25:09.501-0700 [ERROR] tasksmanager: error applying task: task_name=config-task + error= + | error tf-apply for 'config-task': exit status 1 + | + | Error: GET https://api.github.com/user: 401 Bad credentials [] + | + | with module.config-task.provider["registry.terraform.io/integrations/github"], + | on .terraform/modules/config-task/main.tf line 11, in provider "github": + | 11: provider "github" { + | + ``` +1. Check for differences between the previous leader and new leader, such as differences in configurations, environment variables, and local resources. +1. Start a new instance with the fix that resolves the issue. +1. Tear down the leader instance that has the issue and any other instances that may have the same issue. +1. Restart the affected instances to implement the fix. diff --git a/website/content/docs/nia/usage/run.mdx b/website/content/docs/nia/usage/run.mdx new file mode 100644 index 0000000000..ef26f96c56 --- /dev/null +++ b/website/content/docs/nia/usage/run.mdx @@ -0,0 +1,41 @@ +--- +layout: docs +page_title: Run Consul-Terraform-Sync +description: >- + Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the `consul-terraform-sync` daemon. +--- + +# Run Consul-Terraform-Sync + +This topic describes the basic procedure for running Consul-Terraform-Sync (CTS). Verify that you have met the [basic requirements](/docs/nia/usage/requirements) before attempting to run CTS. + +1. Move the `consul-terraform-sync` binary to a location available on your `PATH`. + + ```shell-session + $ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync + ``` + +2. Create the config.hcl file and configure the options for your use case. Refer to the [configuration reference](/docs/nia/configuration) for details about all CTS configurations. + +3. Run Consul-Terraform-Sync (CTS). + + ```shell-session + $ consul-terraform-sync start -config-file + ``` + +4. Check status of tasks. Replace port number if configured in Step 2. Refer to [Consul-Terraform-Sync API](/docs/nia/api) for additional information. + + ```shell-session + $ curl localhost:8558/status/tasks + ``` + +## Other Run modes + +You can [configure CTS for high availability](/docs/nia/usage/run-ha), which is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. + +You can start CTS in [inspect mode](/docs/nia/cli/start#modes) to review and test your configuration before applying any changes. Inspect mode allows you to verify that the changes work as expected before running them in an unsupervised daemon mode. + +For hands-on instructions on using inspect mode, refer to the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. + + + diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index b89ec40984..90156e9bcc 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -794,16 +794,29 @@ "path": "nia/installation/install" }, { - "title": "Requirements", - "path": "nia/installation/requirements" - }, - { - "title": "Configure", + "title": "Configuration", "path": "nia/installation/configure" + } + ] + }, + { + "title": "Usage", + "routes": [ + { + "title": "Requirements", + "path": "nia/usage/requirements" }, { "title": "Run Consul-Terraform-Sync", - "path": "nia/installation/run" + "path": "nia/usage/run" + }, + { + "title": "Run Consul-Terraform-Sync with High Availability", + "path": "nia/usage/run-ha" + }, + { + "title": "Error Messages", + "path": "nia/usage/errors-ref" } ] }, diff --git a/website/public/img/nia/cts-ha-after.png b/website/public/img/nia/cts-ha-after.png new file mode 100644 index 0000000000..5cd256056a Binary files /dev/null and b/website/public/img/nia/cts-ha-after.png differ diff --git a/website/public/img/nia/cts-ha-after.svg b/website/public/img/nia/cts-ha-after.svg new file mode 100644 index 0000000000..088089d205 --- /dev/null +++ b/website/public/img/nia/cts-ha-after.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/public/img/nia/cts-ha-before.png b/website/public/img/nia/cts-ha-before.png new file mode 100644 index 0000000000..fc602de964 Binary files /dev/null and b/website/public/img/nia/cts-ha-before.png differ diff --git a/website/public/img/nia/cts-ha-before.svg b/website/public/img/nia/cts-ha-before.svg new file mode 100644 index 0000000000..546cf54893 --- /dev/null +++ b/website/public/img/nia/cts-ha-before.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/redirects.js b/website/redirects.js index c9c5b668b0..c81aacbb52 100644 --- a/website/redirects.js +++ b/website/redirects.js @@ -1278,6 +1278,16 @@ module.exports = [ destination: '/docs/k8s/installation/vault/data-integration/connect-ca', permanent: true, }, + { + source: '/docs/nia/installation/run', + destination: '/docs/nia/usage/run', + permanent: true, + }, + { + source: '/docs/nia/installation/requirements', + destination: '/docs/nia/usage/requirements', + permanent: true, + }, { source: '/docs/k8s/installation/install#consul-k8s-cli-installation', destination: '/docs/k8s/installation/install-cli', @@ -1534,4 +1544,14 @@ module.exports = [ destination: '/docs/k8s/deployment-configurations/vault/wan-federation', permanent: true, }, + { + source: '/docs/nia/installation/run', + destination: '/docs/nia/usage/run', + permanent: true, + }, + { + source: '/docs/nia/installation/requirements', + destination: '/docs/nia/usage/requirements', + permanent: true, + }, ]