From a1fd32472630b210533e45f074e3074a826ad381 Mon Sep 17 00:00:00 2001 From: Noe Brown Date: Wed, 8 Jun 2022 12:11:15 -0400 Subject: [PATCH 01/57] Updated the default service name to lower case Changes to doc : Changed default service_name to consul-terraform-sync. Service name is used in service registration it was previously proper case, it is now lower case. --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 2576a9fa0f..95993e8690 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -190,7 +190,7 @@ Service registration requires that the [Consul token](/docs/nia/configuration#co | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul.| `true` | -| `service_name` | Optional | string | The service name for CTS. | `Consul-Terraform-Sync` | +| `service_name` | Optional | string | The service name for CTS. | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | From 67144d514e8ee7aa2feb63b0d74194bb244b91ae Mon Sep 17 00:00:00 2001 From: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Date: Fri, 15 Jul 2022 12:35:33 -0700 Subject: [PATCH 02/57] Nia/cts tasks not required (#13731) * update docs * Update website/content/docs/nia/configuration.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 95993e8690..69d8111369 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -241,7 +241,7 @@ service { ## Task -A `task` block configures which task to execute in automation. When the task should execute can be determined by the `service` block (deprecated) and `condition` block. The `task` block may be specified multiple times to configure multiple tasks. +A `task` block configures which task to execute in automation. When the task executes can be determined by the `condition` block. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). ```hcl task { From cccd13ffbe246ac177718aed568e755daad81cd0 Mon Sep 17 00:00:00 2001 From: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Date: Fri, 22 Jul 2022 00:18:50 -0700 Subject: [PATCH 03/57] update docs (#13842) * update docs * Update website/content/docs/nia/cli/start.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/cli/start.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 04534d4a38..4b40b9262e 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -60,3 +60,4 @@ The `start` command supports the following options: | `-inspect` | Optional | boolean | Run CTS in Inspect mode to print the proposed state changes for all tasks, and then exit. No changes are applied in this mode. | false | | `-inspect-task`                             | Optional | string | Run CTS in Inspect mode to print the proposed state changes for the task, and then exit. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. | none | | `-once` | Optional | boolean | Render templates and run tasks once. Does not start the process in long-running mode and disables buffer periods. | false | +| `-reset-store` | Optional | boolean | Use only when running CTS in High Availability (HA) mode. When enabled, this CTS instance overwrites the state storage with its own state whenever it is elected cluster leader. | false | From cd07f5dd047cb0bb95132bdc3355f0b65ef54f51 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 14:11:59 -0700 Subject: [PATCH 04/57] updated reset-store flag and other page content --- website/content/docs/nia/cli/start.mdx | 74 ++++++++++---------------- 1 file changed, 27 insertions(+), 47 deletions(-) diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 4b40b9262e..023e2938a6 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -2,62 +2,42 @@ layout: docs page_title: Start Command description: >- - Consul-Terraform-Sync supports start command for starting CTS as a daemon + Run the Consul-Terraform-Sync start command to start the CTS daemon. --- # start -The `start` command starts Consul-Terraform-Sync (CTS) as a daemon. When CTS runs as a daemon, there is no default configuration to start CTS. You must set a configuration flag `-config-file` or `-config-dir`. For example: +The `start` command starts the Consul-Terraform-Sync (CTS) daemon. + +## Usage ```shell-session -$ consul-terraform-sync start -config-file=config.hcl +$ consul-terraform-sync start -config-file [OPTIONS] ``` -To review a list of available flags, use the `-help` or `-h` flag. - -## Modes - -CTS can be run as a daemon in different modes. - -### Long-running Mode - -Flag: none - -Behavior: This is the default mode in which CTS passes through a once-mode phase and then turns into a long-running process. During the once-mode phase, the daemon will exit with a non-zero status if it encounters an error. After successfully passing through once-mode phase, it will begin a long-running process in which errors are logged and exiting is not expected behavior. When the long-running process begins, CTS daemon starts serving API and command requests. - -### Inspect Mode - -Flag: `-inspect` - -Behavior: CTS will display the proposed state changes for all tasks once and exit. No changes are applied in this mode. On encountering an error before completing, CTS will exit with a non-zero status. - -Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks would update network infrastructure as expected. - ---- - -Flag: `-inspect-task [task-name]` - -Behavior: This has similar behavior as `-inspect` mode for the selected task. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. - -Usage: Useful to debug one or more tasks to confirm configuration is accurate and the selected tasks would update network infrastructure as expected. - -### Once Mode - -Flag: `-once` - -Behavior: CTS will run all tasks once with buffer periods disabled and exit. On encountering an error before completing, CTS will exit with a non-zero status. - -Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks update network infrastructure as expected. +The `-config-file` or `-config-dir` flag is required. Use the flag to specify the [CTS instance configuration file](/docs/nia/configuration) or directory containing several configuration files to start a CTS cluster. ## Options -The `start` command supports the following options: +The following table describes all of the available flags. -| Name | Required | Type | Description |Default | -| --------------- | -------- | ------- | ------------------------------- | ----------------------- | -| `-config-dir `      | Required when`-config-file` is not set | string | A directory to load files for configuring CTS. Configuration files require an .hcl or .json file extension in order to specify their format. This option can be specified multiple times to load different directories. | none | -| `-config-file`                        | Required when `-config-dir` is not set | string | A file to load for configuring CTS. Configuration file requires an .hcl or .json extension in order to specify their format. This option can be specified multiple times to load different configuration files. | none | -| `-inspect` | Optional | boolean | Run CTS in Inspect mode to print the proposed state changes for all tasks, and then exit. No changes are applied in this mode. | false | -| `-inspect-task`                             | Optional | string | Run CTS in Inspect mode to print the proposed state changes for the task, and then exit. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. | none | -| `-once` | Optional | boolean | Render templates and run tasks once. Does not start the process in long-running mode and disables buffer periods. | false | -| `-reset-store` | Optional | boolean | Use only when running CTS in High Availability (HA) mode. When enabled, this CTS instance overwrites the state storage with its own state whenever it is elected cluster leader. | false | +| Name | Required | Type | Description |Default | +| --- | ---- | ---- | ---- | ---- | +| `-config-dir`| Required when `-config-file` is not set | string | Specifies a directory containing CTS instance configuration files to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one directory of files. | none | +| `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | +| `-inspect` | Optional | boolean | Starts CTS in inspect mode (refer to [Modes](#modes) for additional information). In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | +| `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | +| `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in high availability mode. | false | +| `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | + +## Modes + +By default, CTS starts in long-running mode. The following table describes all available CTS modes. + +| Name | Description |How to start | +| --- | --- | --- | +| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | +| Once mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | +| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| +| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to Run Consul-Terraform-Sync with High Availability for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to Run Consul-Terraform-Sync with High Availability for additional information.

| From 909aba1e228efdc4bc6de03cc5d14e38058e6cd7 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 15:07:47 -0700 Subject: [PATCH 05/57] partial update to the nav, revisisions to the usage requirements page --- .../docs/nia/installation/requirements.mdx | 129 ----------------- .../content/docs/nia/usage/requirements.mdx | 134 ++++++++++++++++++ .../docs/nia/{installation => usage}/run.mdx | 0 website/data/docs-nav-data.json | 17 ++- 4 files changed, 145 insertions(+), 135 deletions(-) delete mode 100644 website/content/docs/nia/installation/requirements.mdx create mode 100644 website/content/docs/nia/usage/requirements.mdx rename website/content/docs/nia/{installation => usage}/run.mdx (100%) diff --git a/website/content/docs/nia/installation/requirements.mdx b/website/content/docs/nia/installation/requirements.mdx deleted file mode 100644 index 17f13ff5eb..0000000000 --- a/website/content/docs/nia/installation/requirements.mdx +++ /dev/null @@ -1,129 +0,0 @@ ---- -layout: docs -page_title: Requirements -description: >- - Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon. ---- - -# Prerequisites Needed to Run Consul-Terraform-Sync - -The following components are required to run Consul-Terraform-Sync (CTS): - -* A Terraform Provider -* A Terraform Module -* A Consul cluster running outside of the `consul-terraform-sync` daemon - -Practitioners can add support for their network infrastructure through Terraform providers. Once network infrastructure support exists, practitioners can add network integrations in the form of Terraform modules. - -The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). - -## Run a Consul Cluster - -Below are several steps towards a minimum Consul setup required for running CTS. - -### Install Consul - -CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately. - -To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install). - -For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul). - -### Run an Agent - -The Consul agent must be running in order to dynamically update network devices. To run the local Consul agent, you can run Consul in development mode which can be started with `consul agent -dev` for simplicity. For more details on running Consul agent, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). - -When running a Consul agent with CTS in production, we suggest to keep a few considerations in mind. CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, like changes to registered services. This results in multiple long running TCP connections between CTS and the agent to poll changes for each dependency. Monitoring a high number of services may quickly hit the default Consul agent connection limits. - -There are 2 ways to fix this issue. The first and recommended fix is to use HTTP/2 (requires HTTPS) to communicate between Consul-Terraform-Sync and the Consul agent. When using HTTP/2 only a single connection is made and reused for all communications. See the [Consul Configuration section](/docs/nia/configuration#consul) for more. The other option is to configure [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the agent to a reasonable value proportional to the number of services monitored by Consul-Terraform-Sync. - -### Register Services - -CTS monitors Consul catalog for service changes which lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent either by loading a service definition or by HTTP API request. - -If you are running Consul in development mode, below is an example of registering a service by HTTP API request: - -```shell-session -$ echo '{ - "ID": "web", - "Name": "web", - "Address": "10.10.10.10", - "Port": 8000 -}' > payload.json - -$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register -``` - -The above example registers a service named "web" with your Consul agent. This represents a non-existent web service running at 10.10.10.10:8000. Your web service is now available for CTS to consume. You can have CTS monitor the web service to execute a task and update network device(s) by configuring "web" in [`condition "services"`](/docs/nia/configuration#services-condition) of a task block. If the web service has any non-default values, it can also be configured in `condition "services"`. - -For more details on registering a service by HTTP API request, refer to the [register service API docs](/api-docs/agent/service#register-service). - -For more details on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started). - -### Run a Cluster - -The previous steps of installing and running a single Consul agent then registering a single service is sufficient to meaningfully start running CTS. - -If you would like to run a Consul cluster rather than a single agent, refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started). This will walk you through the steps of running multiple Consul agents and then joining them together into a cluster. - -## Network Infrastructure (using a Terraform Provider) - -CTS integrations for the Terraform driver utilizes Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver of CTS inherits the expansive collection of Terraform providers to integrate with, and with release of [Terraform 0.13](https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-13/), this extends to include providers written by the community too by using [provider source](https://www.hashicorp.com/blog/adding-provider-source-to-hashicorp-terraform/). - -### Finding Terraform Providers - -To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers). - -### How to Create a Provider - -If there is no existing Terraform provider, a new Terraform provider can be [created](https://learn.hashicorp.com/tutorials/terraform/provider-setup) and [published](https://www.terraform.io/docs/registry/providers/publishing.html). The provider can then be used within a network integration task by authoring a compatible Terraform module. - -## Network Integration (using a Terraform Module) - -The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module along with how it is configured within a task determines the condition under which your infrastructure is updated. - -Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. - -Continue to the next page to [get started with configuring CTS and how to use Terraform providers and modules for tasks.](/docs/nia/installation/configure) - -### Partner Terraform Modules - -The modules listed below are available to use and are compatible with CTS. - -#### A10 Networks - -- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia) - -#### Avi Networks - -- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool) - -#### AWS Application Load Balancer (ALB) - -- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia) - -#### Checkpoint - -- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia) - -#### Cisco ACI - -- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia) -- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia) - -#### Citrix ADC - -- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) - -#### F5 - -- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia) - -#### NS1 - -- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) - -#### Palo Alto Networks - -- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia) -- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia) diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx new file mode 100644 index 0000000000..31e8ac186f --- /dev/null +++ b/website/content/docs/nia/usage/requirements.mdx @@ -0,0 +1,134 @@ +--- +layout: docs +page_title: Requirements +description: >- + Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon. +--- + +# Requirements + +The following components are required to run Consul-Terraform-Sync (CTS): + +* A Terraform Provider +* A Terraform Module +* A Consul cluster running outside of the `consul-terraform-sync` daemon + +You can add support for your network infrastructure through Terraform providers so that you can apply Terraform modules to implement network integrations. + +The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). + +## Run a Consul cluster + +Below are several steps towards a minimum Consul setup required for running CTS. + +### Install Consul + +CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately. + +To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install). + +For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul). + +### Run an agent + +The Consul agent must be running in order to dynamically update network devices. Refer to the [Consul agent documentation](/docs/agent/index) for information about configuring and starting a Consul agent. For hands-on instructions about running Consul agents, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). + +When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitors a high number of services. + +To avoid reaching the limit prematurely, we recommend using HTTP/2 (requires HTTPS) to communicate between CTS and the Consul agent. When using HTTP/2, CTS establishes a single connection and reuses it for all communication. Refer to the [Consul Configuration section](/docs/nia/configuration#consul) for details. + +Alternatively, you can configure the [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) option to set a maximimum number of connections to meet your needs. + +### Register services + +CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. + +The following HTTP API request example registers a service named `web` with your Consul agent: + +```shell-session +$ echo '{ + "ID": "web", + "Name": "web", + "Address": "10.10.10.10", + "Port": 8000 +}' > payload.json + +$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register +``` + +The example represents a non-existent web service running at `10.10.10.10:8000` that is now available for CTS to consume. + +You can configure CTS to monitor the web service, execute a task, and update network device(s) by configuring `web` in the [`condition "services"`](/docs/nia/configuration#services-condition) task block. If the web service has any non-default values, it can also be configured in `condition "services"`. + +For more details on registering a service using the HTTP API endpoint, refer to the [register service API docs](/api-docs/agent/service#register-service). + +For hands-on instructions on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started). + +### Run a cluster + +For production environments, we recommend operating a Consul cluster rather than a single agent. Refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started) for instructions on starting multiple Consul agents and joining them into a cluster. + +## Network infrastructure (using a Terraform provider) + +CTS integrations for the Terraform driver use Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver for CTS inherits the expansive collection of Terraform providers to integrate with. You can also specify a provider `source` in the [`required_providers` configuration](https://www.terraform.io/language/providers/requirements#requiring-providers) to use providers written by the community (requires Terraform 0.13 or later). + +### Finding Terraform providers + +To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers). + +### How to create a provider + +If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registery so that you can use it within a network integration task or createa compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: + +* [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) +* [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). + +## Network integration (using a Terraform module) + +The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the condition under which your infrastructure is updated. + +Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. + +Refer to [Configuration](/docs/nia/installation/configure) for information about configuring CTS and how to use Terraform providers and modules for tasks. + +### Partner Terraform Modules + +The modules listed below are available to use and are compatible with CTS. + +#### A10 Networks + +- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia) + +#### Avi Networks + +- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool) + +#### AWS Application Load Balancer (ALB) + +- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia) + +#### Checkpoint + +- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia) + +#### Cisco ACI + +- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia) +- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia) + +#### Citrix ADC + +- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) + +#### F5 + +- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia) + +#### NS1 + +- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) + +#### Palo Alto Networks + +- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia) +- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia) diff --git a/website/content/docs/nia/installation/run.mdx b/website/content/docs/nia/usage/run.mdx similarity index 100% rename from website/content/docs/nia/installation/run.mdx rename to website/content/docs/nia/usage/run.mdx diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index ae5624a0d9..d89223189f 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -755,16 +755,21 @@ "path": "nia/installation/install" }, { - "title": "Requirements", - "path": "nia/installation/requirements" - }, - { - "title": "Configure", + "title": "Configuration", "path": "nia/installation/configure" + } + ] + }, + { + "title": "Usage", + "routes": [ + { + "title": "Requirements", + "path": "nia/usage/requirements" }, { "title": "Run Consul-Terraform-Sync", - "path": "nia/installation/run" + "path": "nia/usage/run" } ] }, From bd5364638d42b12415233f6a1670c690ab7abfc5 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 16:44:14 -0700 Subject: [PATCH 06/57] add HA usage page and small revisions to standard usage page --- website/content/docs/nia/cli/start.mdx | 4 +- website/content/docs/nia/usage/run-ha.mdx | 184 ++++++++++++++++++++++ website/content/docs/nia/usage/run.mdx | 13 +- website/data/docs-nav-data.json | 4 + website/public/img/nia/cts-ha-after.png | Bin 0 -> 28850 bytes website/public/img/nia/cts-ha-before.png | Bin 0 -> 29219 bytes 6 files changed, 200 insertions(+), 5 deletions(-) create mode 100644 website/content/docs/nia/usage/run-ha.mdx create mode 100644 website/public/img/nia/cts-ha-after.png create mode 100644 website/public/img/nia/cts-ha-before.png diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 023e2938a6..801b14ab6f 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -28,7 +28,7 @@ The following table describes all of the available flags. | `-inspect` | Optional | boolean | Starts CTS in inspect mode (refer to [Modes](#modes) for additional information). In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | | `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | -| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in high availability mode. | false | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | | `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | ## Modes @@ -40,4 +40,4 @@ By default, CTS starts in long-running mode. The following table describes all a | Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | | Once mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | | Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| -| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to Run Consul-Terraform-Sync with High Availability for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to Run Consul-Terraform-Sync with High Availability for additional information.

| +| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx new file mode 100644 index 0000000000..50e8dda098 --- /dev/null +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -0,0 +1,184 @@ +--- +layout: docs +page_title: Run Consul-Terraform-Sync with High Availability +description: >- + Improve network automation resiliency by enabling high availability for Consul-Terraform-Sync. HA enables persistent task and event data so that CTS functions as expected during a failover event. +--- + +# Run Consul-Terraform-Sync with High Availability + +This topic describes how to run Consul-Terraform-Sync (CTS) configured for high availability. High availability is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. + +## Introduction + +A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process if it is configured for high availability: + +1. The CTS cluster promotes a new leader from the pool of followers in the network. +1. The new leader begins running all existing tasks in `once-mode` in order to process changes that occurred during the failover transition period. In this mode, CTS runs all existing tasks one time. +1. The new leader logs any errors that occur during `once-mode` operation and the new leader continues to monitor Consul for changes. + +In a standard configuration, CTS exits if errors occur when the CTS instance runs tasks in `once-mode`. In a high availability configuration, CTS logs the errors and continues to operate without interruption. + +The following diagram shows operating state when high availability is enabled: + +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.png) + +The following diagram shows the CTS cluster state after the leader stops. CTS Instance B becomes the leader responsible for monitoring and running tasks. + +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.png) + +### Failover details + +* The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. +* If failover occurs during task execution, a new leader is elected and will attempt to run all tasks once before continuing to monitor for changes. +* If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. +* If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. +* If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. + +Note that driver behavior is consistent whether or not CTS is running in high availability mode. + +## Requirements + +* Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. +* CTS Enterprise 0.7 or later +* Terraform CLI 0.13 or later +* All instances in a cluster must be in the same datacenter. + +You must configure appropriate ACL permissions for your cluster. Refer to [ACL permissions](#) for details. + +It’s not required, but we recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. + +## Configuration + +Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](#) for details about the configuration fields for the `high_availability` block. + +The following example configures high availability functionality for a cluster named `cts-cluster`: + + + +```hcl +high_availability { + cluster { + name = "cts-cluster" + storage "consul" { + parent_path = "cts" + namespace = "ns" + session_ttl = "30s" + } + } + + instance { + address = "cts-01.example.com" + } +} +``` + + +### ACL permissions + +The `session` and `keys` resources in your Consul environment must have `write` permissions. Refer to the [ACL documentation](/docs/security/acl) for details on how to define ACL policies. + +If the `high_availability.cluster.storage.namespace` field is configured, then your ACL policy must also enable `write` permissions for the `namespace` resource. + +## Start a new CTS cluster + +We recommend deploying a cluster that includes three CTS instances. This is so that the cluster has one leader and two followers. + +1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/installation/configure) for all configuration options. +1. Issue the startup command and pass the configuration file. Refer to the [`start` command reference](/docs/nia/cli/start#modes) for additional information about CTS startup modes. + ```shell-session + $ consul-terraform-sync start -config-file ha-config.hcl + ``` +1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. + + ```shell-session + $ curl localhost:/status/tasks + ``` + +## Modify an instance configuration + +You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings, when high availability is enabled. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). + +1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: + ```shell-session + [INFO] ha: acquired leadership lock: id= + ``` +1. Stop one of the follower CTS instances and apply the new configuration. +1. Restart the follower instance. +1. Repeat steps 2 and 3 for other follower instances in your cluster. +1. Stop the leader instance. One of the follower instances becomes the leader. +1. Apply the new configuration to the former leader instance and restart it. + +## Modify tasks + +When high availability is enabled, CTS persists task and event data (refer to State storage and persistence for additional information). You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes. This is to limit inconsistencies between the state and the configuration that can occur when mixing methods. + +### Delete and recreate the task (recommended) + +Use the CTS API to identify the CTS leader instance and delete and replace a task. + +1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: + + ```shell-session + [INFO] ha: acquired leadership lock: id= + ``` +1. Send a `DELETE` call to the [`/task/` endpoint](/docs/nia/api/tasks#delete-task) to delete the task. In the following example, the leader instance is at `localhost:8558`: + + ```shell-session + $ curl --request DELETE localhost:8558/v1/tasks/task_a + ``` + + You can also use the [`task delete` command](/docs/nia/cli/task#task-delete) to complete this step. + +1. Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. + ```shell-session + $curl --header "Content-Type: application/json" \ + --request POST \ + --data @payload.json \ + localhost:8558/v1/tasks +``` + You can also use the [`task-create` command](/docs/nia/cli/task#task-create) to complete this step. + +Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. + +```shell-session +$curl --header "Content-Type: application/json" \ +--request POST \ +--data @payload.json \ + localhost:8558/v1/tasks +``` + +You can also use the `task-create` command to complete this step. + +### Discard data with the `-reset-storage` flag + +You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/options) to discard persisted data if you need to update a task. + +1. Stop a follower instance. +1. Update the instance’s task configuration. +1. Restart the instance and include the `-reset-storage` flag. +1. Stop all other instances so that the updated instance becomes the leader. +1. Start all other instances again. +1. Restart the instance you restarted in step 3 without the `-reset-storage` flag so that it starts up with the current state. If you continue to run an instance with the `-reset-storage` flag enabled, then CTS will reset the state data whenever the instance becomes the leader. + +## Troubleshooting + +Use the following troubleshooting procedure if a previous leader had been running a task successfully but the new leader logs an error after a failover: + +1. Check the logs printed to the console for errors. Refer to the [`syslog` configuration](/docs/nia/configuration#syslog) for information on how to locate the logs. In the following example output, CTS reported a `401: Bad credentials` error: + ```shell-session + 2022-08-23T09:25:09.501-0700 [ERROR] tasksmanager: error applying task: task_name=config-task + error= + | error tf-apply for 'config-task': exit status 1 + | + | Error: GET https://api.github.com/user: 401 Bad credentials [] + | + | with module.config-task.provider["registry.terraform.io/integrations/github"], + | on .terraform/modules/config-task/main.tf line 11, in provider "github": + | 11: provider "github" { + | + ``` +1. Check for differences between the previous leader and new leader, such as differences in configurations, environment variables, and local resources. +1. Start a new instance with the fix that resolves the issue. +1. Tear down the leader instance that has the issue and any other instances that may have the same issue. +1. Restart the instance(s) to implement the fix. diff --git a/website/content/docs/nia/usage/run.mdx b/website/content/docs/nia/usage/run.mdx index d4746eb03d..a90cfae1af 100644 --- a/website/content/docs/nia/usage/run.mdx +++ b/website/content/docs/nia/usage/run.mdx @@ -7,13 +7,15 @@ description: >- # Run Consul-Terraform-Sync +This topic describes the basic procedure for running Consul-Terraform-Sync (CTS). Verify that you have met the [basic requirements](/docs/nia/usage/requirements) before attempting to run CTS. + 1. Move the `consul-terraform-sync` binary to a location available on your `PATH`. ```shell-session $ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync ``` -2. Create the config.hcl file, all the options are available [here](/docs/nia/configuration). +2. Create the config.hcl file and configure the options for your use case. Refer to the [configuration reference](/docs/nia/configuration) for details about all CTS configurations. 3. Run Consul-Terraform-Sync (CTS). @@ -29,6 +31,11 @@ description: >- ## Other Run modes -CTS allows you to inspect your configuration before applying any change and to run in once mode, meaning that you can verify the changes are correctly applied in a test run before running it in unsupervised daemon mode. +You can [configure CTS for high availability](/docs/nia/usage/run-ha), which is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. + +You can start CTS in [inspect mode](/docs/nia/cli/start#modes) to review and test your configuration before applying any changes. Inspect mode allows you to verify that the changes work as expected before running them in an unsupervised daemon mode. + +For hands-on instructions on using inspect mode, refer to the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. + + -To learn more on these options check the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index d89223189f..eedbd4d9be 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -770,6 +770,10 @@ { "title": "Run Consul-Terraform-Sync", "path": "nia/usage/run" + }, + { + "title": "Run Consul-Terraform-Sync with High Availability", + "path": "nia/usage/run-ha" } ] }, diff --git a/website/public/img/nia/cts-ha-after.png b/website/public/img/nia/cts-ha-after.png new file mode 100644 index 0000000000000000000000000000000000000000..5cd256056a5de45ae101325cec97729dbad39de4 GIT binary patch literal 28850 zcmZsD1yqz>7w#~0r*wmaG|~-{k^;ieNOyN5-5t`3ba!`ygd&Y}Nq5J6@vHx?du3b> z@0>Zg&#vd$hAAmXp&}6?fj}Ts8EJ845D4rC0zru&zyVJfM@OGOfO1rp5(AZw5&s5$ zQ8I?en8?e6UIFhBKwu~=5cG2s;0F|94ubjf9t5HR{sn=cbD;kFGza|8r(nMv=zrcr zi99#-9`i^9I?*s!g*ZXv<@gM3t(iX<*%}x#yII>kHvkE^@d0nGjh#M_yIEV=IP$p( zQvPYd2fTlN%tA^2r-_rLASFaziTsVNgE2W5Gb=MIr4SN1Ik|v?kqMu&xa8ko2YwTz zG;?yY<6~iQb#-NSea&p^V9LVA%gf8c%Fe>h&IGhza&)(G`ryW7<4E;aC;#*#ZtQ62 zU~cDRZfisS-0ueiTW2RhO3LSf{{8dUJe|x<{u{~0@$Y2;3uJlzgoTZnmF3^Qfv*ZY zKjl+&FgFH9e(qn0P2f+<|LfV`;|Q=k5B@)s`D>+ro&rl1LK0y4ciV)J2y|EBK_C&3 zjJT+(8`OR#te2`3UY{ZimKXx!;i2xL4pQG+dK}>A5GVXaK~S=C1};-jeWOZFqgLD+ zj4LgVf9&xnNxM9D^~vbg!4^u#=;0XA!?@wqeBG~4$eXvA31KQECCiWPiQkKyR} ze;Q*t(ek&Fn_2k!O$-e z4EdBm{T!>($z;d3s9;jz$)=_7x}!R>b4Tit4-V@Xc9+tHrT(=rgr?%NSkLQc<${90 z$ow3nkHVE;2{Od6Neov5E5T#>R?t0u>7CI31lDcif<(*`|Fv0E*ls1YegX#Uz%bOm z3KN+upAs*zzvtv{=PHBG@FPV!iK276f&)kY%&iC8ZLi?muk<12vj`Oa8~t3BeJx>) zBfLzk)Rcctr*j$+rqEcQ?FAjkFx*OYWGY?AV0xcuDxZz)pN;l=fjH>EItN1t?GI*! z%aJUQKG=Q92g}OxZ-ZYL2sxnI^eU7VDqBOFcH#YS_oROZu!H${4P9@vUlHgqe_!o0TTS-Z1w$r{a5k>k6gIF?qmG3W)NbaydRj=N~rbifBm~i`GF@6gA++8 z|B-BR|4KkigGWQuW&RcB&O-zx(EP91>No!_3WEy}@5-CwSO4`tO9F)?(n6%{68g_p z`oVP500+rMru*{W{-Ii^R;*kF4gSo(a|}_y^Msh2Y_Vsd!4`E!MM_HQt3o{Sp^Z(j zB@q!08`#yGbmeTfG?6#=+ZtP4L>!xl!bg64-~JXk<>zc z*~z=IG`e@KbZQm4)*)@iDM0%rV4!Ou%=W+9(;VMkSl-{CacqnwVXyZ`_4xTi+iwnY zx@GgnyiF>}>DwMv%9o?Jnyp-lrmwUFwi-~f@Xs@j8yh*2at90>|TW zKWx%;?8}>w=uk3Yp?F$Fe>TJR8T-D77gFnz*I>x60iA6s9uLb5ilkNdG}FnjF4CqA*WI)gbhViZ7U3$meO=A z1Lq?qlvy4Jj-MKM)Z)(?>JcLGg;2J~GAEN$EvAa8K6D4Ja=rAvJ&&zDZ8aGsWYlgX zUlfqTXh4K{;nNp^7jl1$&u)Z3$Y~Mi`}BZkI1rtO!7;%k24$AhrLXd*OBw|2b}xd1 z`MMA(1N1~j4PyB;j{YK*09p#JzoNilu}X;?zD($2DLg(q zKSxP!6%Bx+2wZ7(Kl_-sO0@T^BuFi%|EQ8Gv6X{`hQZJI%eah?z&6jT^^1}jI<;C> z&H8)drKKf*>-k!9s^Jud2%2n8$vJjo`>j&$_B$I60YPl|{8eNux9m>gr~BOLa2tsx zht1l$Bk$EKpWEH3lJ%N#p`Iur?nDL-Q&ul7_Fpb`8*dEzcHkhBNdf4#A;UCj2~P70`uPYxZ4C@ycW*S-#@N za=w;cAzO&#%hY(5po%bE`>K5&F9_ogNYQ+NuB@Q z;xWn(KcQD^<-b4YxH5Z^Fqq1g8<%FTVtpd$4%<#!9z1U{<3+HN#!ao4yXh!Riawah z|I;V6C6?Xz?Sb0PUPVX$&iC9pj8d%z=Jt0)F-9c6H--`}2t{uEtGsR;teH5}(bh|! z)uuk@FzS!xRGrLm0qDY ztE=eriB6*()URge0EDh=pIgL-+Y4lzj|vJ3niy~|e5x5goUHZDRvSm0%HGs884%$0 z0Jg&rpEcsot;4D$1JlHrDtrs0OT3==sLhQRnS4|Z0a0PTO?nk%Jd#Y|xbD{8l)knpJ@cL8?_21&FEDg0Qg)(lhpy3PGeQW2Hg{0EEy ziBTvC&|OteX5HU*%1;L))-i9BgMjy>Cj;YK5<^NB+U3<1I>o(is~Z$IH+Rd5zGdbL zv%VG)Tm2u)PANlfT%0)DV9R3k>%CFW`aq2RS}$T96_12b4ZAln1-a#8FA|UAmYiUj zCDuoN*TbJJ{sWN&0Z`CzX3Ghq>D)7F9S?U`xxgjC=hI(&U2$%IK&3KS%w ziFsKK`y=(mjsX#1MeOjtyZmf2nx=W5#H>T;2`DYkPLv;pfGdCklm15|vq|JgfB!NZ z?uzxkSZ#7FTQ-F`>|$@`Q`zBEiFzu#F}uU|aF|R2&02a_K$YVRDYb%}>4b?tn=&>= z3QGt&*|eA)a22}(bEsJng?ZV&%6(#=w_V8*4LFZMri}dGP4;__NG+SJUDiVw@T@si zhm@~N?Wt7h^c0#_XG<4O0Of4l^eMX4-C@;})A;Irwh)i=ZK;r$!n7+}tcWmZn465nGifFbvuWpc+~IaSaH$_nBz9xbUWy+rwSE6=NJ}jpORl-8*X%~h z``Rd4jX?4naWa_*RXz7B>-akcS85W`5Vi5)B!&z7fP;mSn!+h%uq+I(aWCmnS>-HG za}x*+?ZeBkb55jAX{v}h6dXUc5pK1&1)#_m`#vQRV9sT~*#eRkH0MZPmlKrFI zOE8a}6a!1tnF*#5TV%cPzRIL6S_nYmj1dPPHV+Q02k&Hivuk50Zh=kB9glqp3GJ-2 zoQdTyS8i--(kK+ieFMIfArDel9t`k@G582Nsz}YTXl3VxwQ6Mxb7a!2GJr+GWei0j zU?-P}r+(IHHoux*Z1Yt102|g%yOPEN-Ab4gAn-cYj|p&+Yd62woYX&E%;?c+*3zYM zn13vOU-IntL8^@qFAsl3Fltn{w>L)kF*K6k^%ubLxZck8JqG?70A9~v2Jh?J ztD_j;?#`Rp&RPBK__2}#?%aA**suDXtum16U}v?|qZQbz8gSIG~Q9zgiVee@p$Ru(DndMlLDe=L*VaG~DXY`I;qKZY#k z{P$#tM*RMz#m_=aUaB( zWy^k0&=ci4{Edx`sb#CZVc7V5PISpkTKNSL#ukuaI_0mSbLWxNP*hkvg+8@%8GHl< z3fW?7y;;{gJ!w1BMXyN(eIhI1QCSx@zFNHgiM|}&pY?FZ`r(v1gU21q{Kr?(sN$)K zHb=bP2s{=@`-pnE{t;l?xRxA4Hhc+l< z*(AxmWW8Q99macMaLx8(xQ^%5Js#%9`;o2eT(LO^QG;&ZCaf=Cre$8>A1yegV}dVK^?E_;}1B>dR9TmcwD7t3xz5l6g+7vEk-RkZd*UO zXltDZbRP?CM;-9H>Q7EHrE(n7e#;adx4L>ww|7l@L6mbEG8NZ5f=s4X;h+2TSOU9# zl0eJq+Fz=Fb13YVGvaZ6{jL^9zwFleWaBrswtzDiC~pP#SDcQ*zVxg83XgxXQQe$ zvzh!dpHbmts!n?Skt!n@Olor64pKpnol!Et*_Z^g0&c7(PQBF}miyUOP)tnB5^Af* zg$2_iC4?g(&hG<)fbS!*HNW$Ou+^o_LRe;cgT;sUbq+7?+st#@pB}Q?0tK2~_DdJq z&SQdZAMz%bAk78@_xJa#-lwYH9^LuS_7|h#`yvdNA0OslVW~;=?YfC0_LmGdY(>u1 zR9bB!=_J#SPaiC>I^T|`UJV>I`m6^>3*|8GkLwA9R8&;p7M(H{cvz_!I1jT-o~JY* zupzSOwoaH+AB_pNIFoq3Z}}K}hR346l`(gHaCL(*VlZ(WbNi0=0IMVx;vt4?0F}fy z`D$-Fv&FRz0fnioZcXC;hH<_~BByRQGRnh5{!U^!0^N3LdTM(#Q>r87u5J{s%x>+c zj9@)7E=zY(i8^#T-PNex+t#7Rk(kU0h0>d=Yc}s++uw6LTSfK0eYzsj*3nsw8-#ZB z<$iiYZ|2b+E#N=yV`q9e^+ZQ$O&q2BYoZL} zx$L1GR9g4YobN0USthp7RCQ{%`Ji(Dlq`2TXBOQcUEItEX~UBZU`he9~&L-Lp@uVyVavib$r< zTSwX$33+%T7SBU^aUqCZn@35gRcl?w)I2#7cwd^r&TVy1t(YK)XbJIWwjffF!e5;2+0x|qrWvd>JOb7cMT&Si~?7<(sqD(vDpds_heqr zZk?smj|<~Ne+bRp`A!h6BB|j>Gir00fpv@BDN1XrK-?@9o6qedyw!3e z!93;HX~VF-aU1~wtkuVh^w~z2%zE5%Wy{vy)44izMI2TBhl*!CqMG&@~7|Bic|^mWD^7Bki!d2LIuGF zNw|A@fiMQYyKQjhHY97h@KnfESWeR;WAq& z<}{J}1Sd5R%P#ICBK>KUkh{wiRd_1%MFeGlF=8OK?cAvb;dAuTD+AWe(rsQwClx}N z_MNMw&@*9|AX1BsB2^hs?EO_TQ?)-X;Q<{O+PqcP;U`xSj3<6>Jf4P%h(#@e_EIdH z{sYbDg!!PS>mh8Tp#<#BV$~mdV>f4;=e$}Q zn^)U~_nq_>sm$L(3e>CxgJ>=BV@L!7m$$$5l?%2wO=p`bNKLmk)g~!XUXmgRc{AFx zTQlup0zP@J`%A?*;tJii?vH)Qo9u5GF=J0ipL!snoOcQNQ71A9lFcunplm7BgsYH- zfBK%Do3r|MWf$f$)>jt|;j};aMB=*E{cKh{IoD?%oQKfT(k{06Dr)5!|Jpt(E}h6v z7)|PGJbuoAjfc`ddt>B3V2t?W+Ix-gN+5l5s{@GZ-`~oA&k}N+$1%Qa%Ab#oW74y+ z)+kXgh^|j|vY64W*yMLVO`H=DrCR5GYEYRqGR8c){2o$gx3yzms#}fnQZ})3BfH?s z(~78Q&F;)LbB5H#eBFxoBXKY~LAb=(6GB{nEzz86J2mUUZi>073Pu;Vbw7)p?D1iS z?OYM##OgRnw=ziG+YWT=^~m-?VK#-8a=y$Wxx8=>l0Cdp zp#j_1rts7BlXV6!1R5<_;LZ==ZEbD6=1URLNO-YQw??L|V_zA+b=qHgu{Hg4z;6dC z7|;5?r_JxQ_wiDP-?EEtPu^JYRlw$Ra3B3r_BmDPF9K#yny)jY4ehIN%0t3ucWBhT zqBfw5;9Jf2$!zvFA8n&M?SgM~x?$htyoYi0NiPXX3zgBp*Av&!_=45Drkw8p`_uL( zhWpt@kQae$>~Ai_u`vVZrV3ZREFo{rf$fn5DMi!@HA|?MFm<5a(5uG{_4G!gd9Bin zY6AV~3de~uR~BoTQjPUD)^UmQti$)Y9XloTaW_X*hFx#td`95F8p0;?bX4P0#i~;V zV)^6*1|fb{*KU{h1aDIqqxdYMvaCqs$*+PKU)wn-=Jb6yIuty%Z}Z@LEUMl}6I)=> zG~9PJJnzrR^6p@lfERk=d&)f4J6`Z=!x2rUV!Sc*Z-2C;YuSQ+eJ1f0*cnMAJZz~D zztW4AFYPF~SVpFEB%BW~jag&;$}lkWe5gWDR3bKx^fz{n&N-4xpYA*_^kU_e^2sN^ zt`UunGWpZLtw=~rW+RW<_y!L-5wc#H!V;%Ad>z*)i6Qh~<>J5u zlcp24#@OLeN&n9Y^7UFzD2Q;5bLXY+xHPzfX<3 za)*Anhbp31$VMb!klM~=vj)>Pt$28o^I0ahKNeNA8rl1x*;SZv<8~y4rAj&zK3g&q zZ7N6CTX5PzZ>mH`YM@MYdS$?(?Nxv=0uSBPk(U+3O57jnfjhRN5X!(Ct|Nz(b5!)YHuk zriFB#s7l|+Sr%*Fc(0G2w`Y#w^Tn>Ad@waRKasJAYOk)*$5Kx$S7u6l6>3G&Po*F7 zlP&@cv|&0I~ToeWBC zq1}Yv%SEL)H$Wpeyfs2omX4P#!AzWr_^E6EgY z9Sg{n9?s4hj})s{Lp{L?Fj~>5PaO5sn~(8#IyK5Egy&7zX9(>Q*!T0*wD>T@rT)QB2FC`VNdW82)bSeJpQ?2ve zX~F!Gh`Q4zWS;4Xf#2bVVju@>>5TTgS$80G(|T!lHqB)9UT9XRT#E&Qm-S1f{A=Rm zMnXu;vp+LGllB`2RfbL>v0ywcOc<9Lp%y><%+$WsOTMfxiJVzzqwP0}I&TbQDZ&%o zJvn!$S9tZ=g*G2C$&_;89cgOoczl9ECovE4^-|Oh8vl|RYYK-|X$%BA^#UGN{D`q4 z880gxub92LEKa^ijIv{#aPL1!m6qLFMk$>2D6T72y1hFnY%pvnAB+=bsXv-83ewq3 z<2*Ni8Lu>EnRJzJIMaN-4`si5zZKOHU2!J^ty-uH`|IvpGOD?*PDOu6z17SEQBtm%3-t=h75Lq*S(!H4cfYG3 z_pk{^Pf7WBtJWm9Gy85Ju&Tr+ypq_wAaUC+|46AZ9>G;(*QEl0 zUXcHjPcuOdo7Eq*D^{czA(25vjaILdpAh!ubpG;g83po#` zL+TbI zH*VYRhhl}=_-441d|3DcHY1mFK60>9=sjoG!$A)qLJJ;{)+w~xNglex#^7f>e)?d1@B-_e#wj-P$Oc627*Z1d*q)-sCQxeA0$Y6c{imA|`e?r+d^wo6626N0X z0#WD8Bv1ff8bGTjz-E=0wJ(u#x+Vx3cxyp~u2xp9#?{}~fsn)EEtkd$P0Z}>j|X`4 zmF;bK4O0M#qSn^{vweX23aZ`5muTTY>NKO`(1pVB>nFml|AVbsg^I`_h}eLuJ`?%L z{|~e(3#9aFVI-3O2Oi`S2QFtYT*TkaLUq8^$k*Sy{~x6RTwNfie4W*Zlr8ZVRQ#7p zae_7+$u;e(Usm_&NiNfhYPmY3(D-3gsFzEloF{Gm9hXt3MUzD41zBCmRHa!{!_iV# zf967*SY3-`-!LvydeC3$9!SvvR;RY_U5!I}OqM`&+Ne-pk5mlCI4|k_()EW<*Pbe` zX#rKb_x%QCzxf>y;@vznv_T4){6vQNa(?N2h^xb0hR1e_Mpaxj2UQZxafIJ4$+?A&UX|G=!f-gGn~^by}h`~ zdFQsW9x!nlW~^8q@}gI9{@ysE*3SF$7@JNtjRP%B;Crr$eNkPOqZvk!k7Cu?Zaaw-*~N8K#bB=gbCl$_KBg_;+Esr8**;;lzWgn zLsEa2Z_JJ*9V`i(RIudT1Ut$%Pndjft&ypEAQX=jD zvC9R3KiXfk8m(!1=duf@7}JZ-b?C`-9PR}{)N(lzIo#)U@*hp?Z1Gs<<6_eE4-y{l zp8m!E-S%f?=dfDo6EZV%IsVT!Zo;Xeigop;HeKP0xmqZV*T#~o_pso}kqF&4@CH5g zOsO~P;#NAt;CTPPQSCK$05fkCxW(M}{Qfb>&f-o;j*t@$)ncmuh4cB(LPX)U+^wj( zW9$+Aw8WxH?{Ve~9!J8VDU*m7doys!SGRgdq|WK zovZ$SLQ)HE#bo&MQ-xJgmx*}f1xwyYR)f>K(Avo7wE1)J(bY8&-Q+KhhE3k^{UfBoP);Bxvwsr+c_iA5Q#*qHn^mkwVe#NTpMm# zy~1vz*M5Uei}Co>>m>;OY9D(xzZ2Q(&uw4Dup3Q*(xroUHvwD_fOTO8+}xo0hzUNs zbJnpE^GN|-hc7NugpFPYud)O@i*@R8n69DXX=TD^d@8=Tyuo8C4vr!UP#qRCMH1~x z>-E-b5%PJoM)OJKLZn1H+Z?ujTbylNP$h?W#Oo$v3zAt&5<*m& za^;+I==Ly=9x&h&vN$K!{Po6q2;6q_hB0Mq?8xQzXoi4))D+hO%W>9q9Z%aR7 zx$G}Td>wAkTgqqxPYxr0;%l$a?%xryU7Q&1#MqNO&U*?UL8s|3>#iI0k69KcrhTg3v1 zYsV`!dw+Z;Z!jAL=t--I?BTKs=5NjVN6Lu+49N^SpqC~dI2HuAyRlD=Wmf?QG5AoP zq_#kA`q*q$Ves{veT>TOyOA{U+&aXKq2@Pk^#`dAizcHqQXS=PzY{1lLelI)@yZUR zIw%Fg?UrOG*s}VodAgB;Gv`|{c2PmW`(fVhT+a#L^f@5ljRoySVr&!ZeKRy!h(AL zg1MyyQML^EE0~Xb>(Or?Q@>FqztVhSl${-1XKLAp2J;-)&S5s#m`k>IT)P{5@}w9B zPq0tb;Qa3%TRzJ>m;JQVKLM2pxC+4OSd-!Xrv5rb!2gN$s{^u8n&~`yfKT?10)o(B zZ0dAYE!wCl{{3CGXGE-rACL9_0}ATD2bT|DW&h=k*dfhMOT`}i4-P1e#hL*-@m&xq zBcZWpXi_JW^%U2~m8@%dn_#Kk2YaZTjTdn1RS_3kT+zd@UVglGH2iBGmBQd)48N1t zd4ZmN_jW7^@D6edccK|wBfm+;U9 z$dgZ0=m`S|_YbVT8k7LcMvseElF`Ir5)rsvJw4Dsa#w7oTqln8gIGXa;xjw2lEWX8 z33gFf4a4yk7QHpkT)EJ2uUS*%+h4 M{rcaQNHJhoJ%E>{tgIF)-RyNtrfa)>os zZ%s6u!V)uIXPK+t^^wQ@lp2>=8zF2w{v~qz!BvzXY+r4p zqF@exLvRY6>ui}c7d2x#?eNW~I2gbPYpkmSn56ZEY$hKkz`BXjs4<~X$(N(3^g11) z<#E}^_qo}`13*-9sW4QTd3l#0I)=$)nGF<{R?1bt@NM^qfGBypRDhh zuK?iR7u)4_4-X!#dMik!zG$VB{Y}(SSjn-*SC||R6IR7e>U*P05fdx`s0sv<9=X;F z^*A55;`-i}PQm7c)*BuC42k|kuRXVa;x-DkcxMi4bD0^i(DT$A@QLVnE!)xKjkJ=O z&{@OU(0qVs?sXcFul)$)alYfvX*s=vz5xL8V(l9}NDi!C2{iJxAV84Vy&HX>9xJPl z)BF)42|1BYW9c7zLeMSuY#^hK_V)gZR8Nn06}srNwdN}Oi%qXf8Cxa69&>(>d%5nT z89m>iZ|neTLW_md&cWdeK;iN>cn?KKB%tgyMILyK>~@KDRZ!ZuX<}&adrBxUXLtQ8 zKSr?N#B9iBk&pvjeL78^|GuBfM$Awa1Y6)Altk_`ZmFmfI% zyl;OOc<@P12cnYj;qW?a_(7nJn@vQ>< zva`v6tQ_2@`7(`~eB3iKuVepuF#LhPS6n(H&uNg5K`(-vr>7+a{&_n>T1bVA{`t=L zou#%GUByeW-~P6Va9mW1yJhpb^Fe=z&E0Ax0A}Y02bXCw+Jxne-{~FnNZ* zX4KB0ci!dNg=YOq2Q1d15E>@gr_ce0tM!8tZnx7J;kcpxT41k55ki-_Pp)WvRshVu zV#YV@TC2^&IkjUe00;AMq($$E&?XIEHfNf(*+?EWEB5v0S%n_3|Q(sCE)NiO;k=KCu>cK(cx04dLk5IU)`ADpgfmWDy^PGJvX z@!7J<-YpAmc}?~^274ZSv6I7z&nw$No}a1I@CTWf3gtK=DjR6l1S)Tz-`HA<<<^44|_^l_IA6kNO{Zrek_ z{vLO&FY!t4Kn7`vihRs(bEdTvbMHr9L|<>V>e=o@-<3tp;etB+xK=}VYH67PLquX0 zm$hcwqsn53YLT*pRsfT7<*VM4d!1`59?UO1EQzmKKu;YlT;jn`Qur^ki@ed^t_VKe z(UFeNn}s8tFbeZvc`LsXZw1!!{aA^zSGE%X{*{`Yev42vx}6xL2IgF+{UdaJ;=BJvw-Hj^UT9lcQ+-^wf$PEAoEuT3(Bj^BQ)cPb(AJ5MpHKTrLdBHB!E->sNTIGAQQr!5I( z23P#zODQENGr>J@=dGCn$EFd6JrU0!rAm7xoH^Ap_95N3%GjT zCT<*PI{JZwW-Fmu>QP6kSJw53$E6v(-8jaYvx`6N++cfMgkosoItq+5XK_!129}(3 zyu!nCO;2dZCp)V>wF7g^GmGedm1#Awnta1&y6?SdEQL=TH3dk0ov+eGG7_kfzVknY zD8A|CZKf6J#_Hk@A7ryT+&m4E-g|9ZGLI=h7NdSJ5D~C6FaNC#k>Sl(WTv&U4mMZT zcJK8VyuFyfX{-Lf{VTs!gh4(74V717N-k+5X`R4s?s5785|C0U2P^D$3|bR=>jF}_ z?>l7GlmM+AEMT;G?UAuFb8xeo^rclN@(!(O7P#uzDX+1yBe#+~X0wR*81RZQ`q_{b z7s0||g^Gd?4-Xq%4m1-jFkH#2U8IKfD^bDK3;cXi*D|{HXrm(FErdaOF@T^u(=rl( zB+6?p3CLmGGY=)_I04lpe^N~7ZvVBT1p62gOes6;eSc(~V!qY#qBl-y1Qd(z>f9x^ zT}7%etI$eQML0j{_0l<|%vrd!b_Y&|3E-?D$Z<9zW?Az`^XKBF#|PeIIN2FCs=vZ8 zNtL_G>kx8pQ2jE=XBCwqh(OSC3)5sHAc<8d?LfIcs?$zWgW|u2k;`6&O>HWX=0j`vuUQ>a~Ih@{EN?~<%`zKy>h>PS9n=QA;x6aIW$S1d_BsE4x z*h6JqXHyp~$1Og3wm9z9kFj&L0K*CkK}ICsRD5s<9-wUQUBK7yfRc*{Rc7lf@m{?z zhLK@F)7?}>sDhFBX<+s$WV}i-2VnW~cwRCFqY~wNgTjMMeteaQofMW6&A&7*OU^)L zQt<9W|LP?Zv13Z^e~j=IprEE=FhVNHHR~+$VqSbpq{n_HdI{TZXq(&f_W}uKU+`7$1$JeZ19nB^$LWKro({W@p)%+7oyF*Pr@Byh0au`AmGtkNUOJ;%% zXw5)>#tmFwzZVQ@fyBS4ZHPCVFx*G=TVRZkAPR1nBau9hfTd2v^)zU@S&tDupB+i5 zftUdSfayEo*m~J0rWi&j%#Q~4$>XS1pkaca9Z!!fS@t={5sag$vbH2ph4W8F533kH zPa9S%*90o!C9YYhBNo1XVZ{#X;A!5khGt^4C~iv9MP(r9UVq z6nXu9a)ym7h5YY)SWY&gDt@e)Xx~CQcquHLl(#!foG!k?poj)z0#*ii*=PE_hs{~G zCR==T!Tb?}-rQeTAcSW)hBImJp)1u2mRL#hY~@^%`7&I-ic z3j$;7t+)O-z6V~v?!JW~Wq_i4%peYk%)8LS^oXi1m(dCzwcy2kgpX z5KaaQx?bn6Y&CNCD-Vo<0a2jpOubQku!2lKu!dY_=Kal2s@qKP*W1` z)kpgxh$RsNfB_<8cKV)Gel7@$V_=EV>#wWFu7gz^>xeRoRS`b)>JYy}Kykateku;# zG6iJ*IqV+6a4RA{6(;WBOT9*BLh`r&k6F&`o>paz` zPXsmVrH&5eA?$eCXT`h$SNR9`WP&r6(0cpA5vDDQ=}(nxFNRJ3>beBuIL>+YgEd%X zhIDi(P&Lm{BTuk25a@Fa#^60DE$wFO&`?0LT1%LEjMto!$GvHfr0X$8k9=iLRz+Fp zFGGg!=htz&c-dn1_%%Og^h~9IbI7#SF}zna_vSe!N=C#N1GSHfL`Qi(?KECZ9t-|C zIFWtKSh8(QR4B!%7&oADY0ZESs}ji{q6KZ1v{Z$K6-otT)Zpl+>1fSG~h4sh`sJc0D6wT6km$icKpG^RQyVV{klSVo+RYDHgYL&^3S;ka%o>Bn$(v zP2TWrCbM9Vi<8=W`Y|qg=0-NimjATlbAJ78sR?HoEEV&V@f7kY6%SQZtCqY9%!7jhFc1D_?Qr2Dd6T(h{+At7wz$> z`!(W5m;N#g53JSlx+Qj7i^C=opVN*XP+yYccCvOW;*qOhIuE176%SJ?POiI^DE+8| zTtg&aG#R2Dk^M3$Nwep0ky`AW$%yt)zI)e*pUWv(Lk=*1+hLjpFBWnPJz{(>p&}Cex@k>H;)umC+E^ z{&EKyrv;22RU)+vme<`u1K{O;sYl?loJNrd$LT89Y1xlP{I}Y$N|YSPQFYBBo>oq+ z#FDp{as4t`0Fn{5k)(ozfO|3!M0pN^{2D z_dePYZ;UD|1uu=`R<5QMK*_nR?iU#IaCQgAL#8ZOs^`fs_U@PMF6KF@Wz%q?h`4^Z zFOO5tG#t!EX8;Lw@hXEOz51c65w-H?;NfGPpkXjdp;Eb5B~N=80r-hjI2btEPfP;7 z?f193T%K%+gRhrNDMow&&QTB`f*?$`gDxgFxJ18X-z0fXJL*7Id;cFi`j| zccS<`0~2l@N^l`|Ec)s^>hAiF%@O~ga}pjYzVT-^~OZ;idljOni5f-?yDtN zHR8jCEJI&q5V`qTktqDF3i0_LN)Z$`H zdy=X=HC&g^hI*(4`E(q%w|T6$tWqtVv>!0S|F(nV^3L40PwQp>CbG3{~fNC6~OXKva@9)*r=DoeA%8e zTh_-{R|5cDcm7p8zI6jcHNEzp_z~(6(qa{4;^Ooq%CvQhwjbkUKDo;lYtQQasT|S> z8DP47P?SsKlF-tgDbp@nk0ud-F*P-fB)xmncz_$R(sp-sa=IzIzgR;wQ?7@eynWc_ zIos|CADziViqB%mfICn!vm}_#94aCCgY^BAo^k6->85zK(sOQ`rJv5B&OBzZmMxcu zh-x9fZ{LuS>EuJCdXOtf1*i7W4(E{prX!L}!6ch1 z4Kl~3Eh*|#JURdUTCj~q#flx4{8aiq|Gf1sPNZKwJS<1fI9t#rzTPG?Qp24;qg5~j*uGs1>}Ta7Rg{HiGTs|>v3M^^ z)FqMyc2%9$vmVD)8gqPv3SG8LN7H2+Dh#j6rPDdF z(|o>zB}Nj($3EUXlYaJTeW!8gbf>AHtqdRSJU_C?gyl>rNr;WBFk8vRz@ zJCRM}dNHh4grM<*yoNiQ!y08=mHzyJ+}!A<=kCK*d)uG}Ihm%b8Qyj7+5<;cR@lW8 z?2T}Op0~QRbcqiK8|1mTd=parl@5zewJ_ENmMXc+e!c&*dWG&hJAU66#IFKpq8-iTUTuK$7jKd(D>q!g6}; zrqz6;p3}TNHq8!Oi_at0qnd&8wCkZTz&=HLLPmFnNv%~cjZZ1pp2kh7@Hp=Um_uxv z1&AKMMU)u60hNB_$wRic=o7+70PPD*R%f$+##MHaa`pZQ8KY#2qVBgPz_hGXKe#;t zxbn@ehadUX%irWXc930BaoH^4SO{wiV0R4(`s1Q7Gc&&$SO1*98I(#(E>@v+q>I6j zGhW7z&%FgNSRRqW)OFkO6nuY7z~)xiPV*Y^)cr#>vFsiiiBKRPs*og5lgjG8=;m#k zW_gjAU0{w$FU)9kDeBXh%%Ib+t=umG)R|3#B&!rm0Q@Et7R>IAn;A>1Zj&M8K^!ra za!*soWwyRGdVP7v25Iu(e4J}bnV!E|p|SHeox(1Wo;W_AQxR}GRhx!w{Q7^zhymbyb-K=u?r8n=O zuKR$$^W$@FmO(;9bTpb>*wL51^w#D38Mso723)~guNRFUUhbqcS-r)iSOCbFk=KV> z0_X0bIxViEx3lHWT5Y1zT=+VC2dJ2~*aZsHW50~wM9KvS`<&X`+O^H2P)&Mmb$=@o zwv0Po+eAhoU?V2;*2ddTeEIXWy@BJr@BCzmP}xcXm3;b<2Cc#}rdSqXIoWqc0SWhwg;~_T6PgEiMbO z_3x9V1!EbA3?=PbSmNZv^P&lxcAJzhS7yrJO!jA+k1Q9Ars?1QpW?nUDyyz*mu>`Z zT2e|t8tHD3Zjcs`Mgi#tMY_911OZ8Dq&uWbknV12_|}cj^SY8 zQE*IBDauV2!sFLwjns{2Hls7S>#R;NUu3OV!1RP(Bj@X#nDL9ar`jw%s#$>Q@PJI$ z|5Y)g#xi-VYaJvGu<~I8dFE#0P@HlEiBpWdjz0nLR_R;^R;e@c^>BcnoPM+>Pa4R z8vD(xYBZNO@o4@>eXM6ad1MZCoMx+UnJdhxyWGwBaOG21qh3TKUjI7TXhoaIi;)O2 zncp>XX5U~n93J9GJVsKchYn( zJ{i|1#2Q8H^m?mbF5wvsQa{ec!PC^WJ~KHQ<3vB(O}DGlf?qZ`A@M(c<3VuL*2jXe zioc)}ECZ>6SBt}~S5Fw`aO+pj`I>`v3{946vntV30FGXQ93siqL+JjxD)ZwLUf4!c z=hjzM-HyE!=xrHo86|mxj|&mcDqNJvA!grb487o_X?rRQy!iU- z8@@bCWKccFgECfM`xwJYY!{(G&cPc`Qh)&V_~g_42B~mv`5Spr}cdzoF6j2ualO`e2|pGFJAxkl5&{ zJW4pj-;}ps@**2fcT0E@duEo;19dXjOwg#F%V3h&d-PkNW4~2tUZ<+gnTAU!4?aY_ z_{%Xq#C=TJ`GF8C1|&x)HIm$};KT|BP_M^^OOk++^(TnZwcdNYY`_BHHji8=TRT=n18 zvSb`*&l1HQXjpcL%n6&vV{}jza3Ub&f?g=)j~nV}$DMl!n;C+`qMvD(()SxV_ROd` ziEYPva#=<@14u+*s|P|yI|(J=?HbiK23~9!hF2g&)O^rdsbxs zt736SboKWFh}Nk<4gHFAvH9*wo&uJT_=s@B1o!We6A-fj8YgTRyg%?b5zH9Hm^`Tv z*60C{DD88ZBlndk-_Hc<#Z{VU!1lE^p&!yLN_EA;;Vd>S_GDS;q60TStd5ElPXsE3qhGS$dDxqNQi(uF*= zl+PwhjTV$Vy6mI<)=+zpGF=S#w>Wh=g+gnkLkcKKgmNQ+zmzTvU9mp-NP1#E;{_=g zP=K<{@d034t99T7G;5~3LNH6R-yokmlDss=qwro1*B<;6Q2c4n==-ltCCn$-+ewRU7Os(j!OVsHk6ofay3s z3tNRC@%vpe`toDxV}k}Cp~H$=v>3S$jW6D8)`KCrn>4`)kl#~F{l{+Zi9kS>6Hg|V4mTB_6dB8hs?pL2a6 zk4!O1(=MLV-*p%5x~IVy{}2=OWCHZ$B_`Li2eQ#74tDLGQKxLHfb;I@|zCVh2`mxm1KUOoTl}J zj%Q_|fgc}zlF2F^E->&kSkLykeUhW5^o|Wtmz7>-pvwO81HTAwex$J1>DMieEFgh| zA?YX6N^HqC{Q%uT32SNebY0rIT&tD1SiJcauB(lL z(xH)`I1d~!@Ox)vHjXrSV%@%+#ClQddf();P5~yw6ll6UpL8qZg4!@PBP^eqywW%- zYY)FL>egtUl(;jVULFRqYK|p4p8C_R;0dKX`!w%K1PZO95?9=?PL;On?r(eud`&;! zG3hZ@*@!szxmw(;52axto)ETt-xzv#XRG^t+v3vn#ld!AWZ5gI36R1n!IId^PCuY- z*_0^_>fd=3PSPCGOfWk;4j#UdK@^sbAdcw6XWWKU%aaXhg-4?DthSxb4MqAlFY9?n`LW_`cDg0WXEy?Rv4^41RcRU`+Sd z1DVh6ad{U%tT^qqoE(+?0AEp5m}@VMTF=ExKfh!!jX~R`XXWz5wc%Zspm)v(gG~Lp zBlC?&+2jHAjlgmBx>vNPTffj(o+<}~Sx}9C)gzHhIQbLX8H!v@yBUVtN0|M7S?Z?| zR%&7?u#0cXElb5{lckAVLDGMjNkJl=xG#Ndx{SQG0!jHCEo}n0b|PdAD&~7$gS7Cnn#zPBX6{w!X&^i&&DDC=x#PUAvibX($ECMS0 z#YT?_Z_#zN@!ic#+veI@0Z*1JnRbbxCW{e^gwkor=)J_vmr!8dLAa3s7G?q$X>ZN&;i}oL7bN|1sRZ=>i3p3{&4g~p>-gapB;rj7 zxDEAB3Ibs#DO6!zF8WH3hm}2rrN?{x-0D5x0IjqPx~B2uDABDu_T|MfxjViZA70uV z`C>Dm*%2#88HpW?3frSd>9D#U5@Jsmwt)mNpg>hCszuqf%UWiD)u`#c&Tcr7nR@Uy z_M6=?X=z*XS)gZ>4R(fO?uS!(5-Q5E2If>Qo_As23k%F2o+nCz;qfPOwVmepd%O<) z9gG9^TK0g15C73ci1u@U?xFuTx4p#;cIfU>Va9Ft_!vde9+t!!J^l>!czwY9=#K#a zYua0Dn;z%(GO7J({D^h9+$r$1CJ%1xixNCIaT(M5=X@S;a%g}k$J38Ojv2zPTX2sn ztzZwr2;n#rne_4GUk^$h)7+E6z*i_wsxh>BqXEB zGVmIUB1bO2`l3=v=*H<3Wze8LxD63)IsRPtTe1)Z%h8HyUwVp8`4a05yE(sfI0bq+ z(|UgbNmHre5((}2-An}z$E|*`zEt?O2ucW~eq}!J`od$JZI{0cE9Cdw$Vz|Y#ZBBW z*`SY3K`4uMHnUD0MmIoatR6XQk!fiCb=TPWV0yc@mZEev^n9*}uOdl9m#Ye?k^qzU zRRM>kPQGwoLEsjx*Gvii;8#u=WP16u;D$^IWGdAh$**d8q%OO&fu{LNRJx|Ht2I5VInz2|emi<6ER5kI?q z=k+g~)8PY5zTZlptjy=ExmlN%KECG94s%fnhsUi$Cq7aW(r4^yAr-#L8N*biE)&}x zrolSbKOj%L1{QqLzp-nE;_)7zyW9xAPOm_koI-9dohdWaMPOb0#vk8gvIp3?Yn|%O z;qk#D<-a2~1zL5zIsBaIN#GJl<+;$%gIL@V-_?uh-fwI# z2TA?=ooF>mZl^zU6jEWigq}G?>{wza0;` zpfrLdt8uy)PH*;-YaYxJU{#}&C)d$Sc}0sYVkmfP=(3c^sD@#~eILO@%xT4bVYTN0 z?Ibu6T{<|3!mt`c?5eRk-rszd(w0!>(5=Ocu;z@)WtYw`^}^kX`*dfN`YoE#UiLW6{=p2(`@4jF~}JF6RfgdqUs5bE*ro)Qf!LF%@24SO6NETIGv?q;cRg z?I4-oX1#xz#WPvjjX_$tFjZ`b3wQui;tp*CMyKPIP1s#ogseu(x#z?gZ6h&V2W%=w zkiR9<3}}~5U?c6%C4({de-XlWhf9_m(?)Z+qY0{{37Dn2Ol#jpE0SH@nU4`9-(Bmw z6*0_pTTApDjpdiS%m=6x-K~D+zp(Bw7Q&n@GU&$MGE3Fcuj;99cchpar8?*VF%S2Z zjJd-=X?}S3oggv9b>)wu{trjuG{RyG)kuaM@xYn9q$@s|#BY}RrDC9d@gVE-z;1wg zHiv?C>4TzwG4Gt6d;PIoMU^clDwECi6T*^+Rm743zs2PzOQ6&dAd4YUijVh+j>@4e zJZoa65$n{WDLW(fCk4mEGxdf7^}Nv}Ger{@91H8>Pwy24YQ!ed_<3Q@tjEOxkH-0j z0#^THv0sSk3YSl8uN)*+dt=m%(63~?U^0bWx!D+-W}KOp587X7p1o{oL1({hgQj^0 z+%+BVG?_$b{AJKx+bl+u+_)H#hq~6p>5MU;V@#9|IJfAhNuLmBzAWqC2w(v9H};O3iV zf0)B7n&Zk*4nh{`qIm^QcX>fXQVgkuLf7NN=svS7xj4QPFQn;R++TlP6#UB`QsxPK zkBdmhE+x3L5$DkT3W3jQ{rPT8P*YP=Ehtb{G?-WNobF#Yxbu~1vPk~Lx^9D zw{CqQXRD&H;L2!nSt*UNJjhw>tSYA;M9m%?AfyZi2o0f&n*CGuf9UD||J%waVMRnl z6o7W0W}${ydk|*1)fgRm;$RNQb;6-aR6U>Ds~^as7}(G<()vKk=cqUm-nW!}iOgX@ zjtDB__EqH*9Ker~!&gQ=qrdrt-hR6`ZV>YZ)$f4Jd* zcL8C^?{EL?>Ouk3ptdSiWj#^mdN5C9ix~!`|0;s%D3{A}hStsf5>S|onn-~pl0?2z z#&E72F7pjYMc+$?KoF}bfs|e{$Y(+M_d`5`dX9db;|!zQ&Bam6b137+m;#zXnv(uP zhejAE34RLa+?K;Ipw5IoeeM+CDO9xzfm0IUs!`+)Qi%X__+$znZFu@cDO28c~;bgBpRIW~AKtK3W( znlrXA`6sa&E=fCRLaR7NJf6(#P*;&0ekQfdk z@Zj!8PzksVr}cc02Qz`ttLI?=S)c$+67EfgIS{yRa{2Sv|84=u)&3V-s3cxZ# zP)s<$$x?zt8I#1`Ffa?%axK5O(|&3}M^bbZ<%ggy>U;gJd^1O2g5--+)%Yrx}cB zp}*kn6W+-lix~T!=cR%G!e5&24Oj!#F^;+@Uv^dzYEU{%ZZsC7u-Y;%Y zM5u))oB>HhW+Huk{l)EV>sE*^9=Lx5dcXJkX<_L7IB(y%ey|7}9bs&@OgYeRG^?a5 z7lL+08W6i|ZCj@G|5xzH0tjlG5Hla>B{p7IP_hG(veI=4u~WUh3=i;m*%mio3=uqZ zwm8t8MUXeXmrG(+kBZjSOnZ6?{~q>0fD-3 zQBcIFG*?!-luhF<+;dJMTM9}8N=2Qm+y82)E`M^Z* zbJ!9>-`v3lHBOV(07N&SSJ^@Q91M@>?A{hcN@cU4RnuaLLD{ zjQ_={i3aUUaF{&!j=!xH7>dq3(0BFWfMCPFjChc-W`;n(5+tFj4kyx|D4?>6rQ>-uHcbai@gDSdwfLc5 z+|raEEFR?WK*w`l_^G$3(sq0Or_VITpFt1p-Yri5t1<`8Y#70}eT_8*R{bwKs7w_) zx%_W!x&Iv$FvSS~N~6*HUxjQK)MKQfbiLb<|N4~yqJPwSg?7PV7k`CN6(|$V*dd{C@4)v~Fx^H9( zZ3(}R;-^sij0C_W5mh@=OKqKDhkyP^k9DWE@c7;}lw9t#A(HaC!O+SlN{U{hxn{U_ z!q2+ycNFU#t>=);{Z?tT*0i$_?qS8t7Oh4kf_a@5NvP%rhgZggeZIr~Zu2*4`A4i2 z+!ruB2Vtd)3C{>qD{bwCeKwMi&bC)}_>8jMXD4|fuH#;Xexz=A zCWGP1XZf#Xg9i_aP2OzYnv`O7O@|!pEGolgk`S_(9{rHjzq@{e@%Y#y=zP#Vpm?^r z@MZ7Zug2djQ^k7aXe3(2UEvJsPx5euZ(3N+P4e*Eh7pKhge~8s56Q!#@k$zZ7HZb7 zm)hhH5(%5pGa_q=mu4!j#9=-CEYaXn@6MCs$-BC8a8TMmUg$30xR9gkq~GA8NO67L zhu^rXcU^7g{Bx;A8`(_f4$#)Amg@{H^Q^Jwj@P9N{2Zq%ox_^!mzL?{3-Gd|Hkj40 zS(}a_5L{eU+FYa({l?HNQ`J;s;6BPlScP@VU+fon0BKvD}BQl^c(UhYgLX zOt=Wol;LFVjv@k>HNAc?uCX4%iX57mnjfq&-KXSmIA9T42rhdeBfa-yxmJb!9)c5O z-}8qgtFj9VBJ=0ArW&x0JvKTQm)cdP7nFGjv#w5*D{bcJ>_!{@DBm0*uS>Crj&((u zv?V+6I{jgi4u4!QcD8dLx-GLAMt%KgagJvp2*EupiKX>4DxXSuU+}BW6VarDixAw+ zqEW32Q@a$s3N3-cwtiuM4?@UubJ(A znfel#i-zhN8VbY+gzLK$AIXk6g+KmUan@~jK}$!Q4FrMnl%9!X`mXD~clAWmBrgn* z8VPeyn==hVL!iAN5wXU|w45^Z?Tcr@Y=+RXiEuz1z4wMw`4uhSqU0noOPP&RjjQ;- z_R#wrglY1eNVQIlcHr_D{ZxZ1t=V?AB(U{#^n?hkB|g~x`v5Z*th^ksvPAu#!-+A{ z$J&LVW$ynDYP`*}f1HrrJ<9EI!z)=}tb7Ik!tPgZ?E>~=((qzN?0CA#@amhw#G%Ty z#aS9W!DW6!-if?n;5?V#Vo!swu+vR-g4R9#+iS~WDSnM`47($3)d@&zbxeB$X z%=N2YS1HjBG2bF*?TzUnmN5hxbV#SRE|794#$;{gb+;yYQ{k}OVF}RTOHjA@* zSF7p=H;HOiYS&ipd2EfQ;sF!jpWz>{;%coFfv5sri z)IEF) z(YOdB!+oJn5>{z-jOg;OFxl|?Zs|I9l6>s9P9HsC-{jMyp)a}llknIZZ`JfWJZE_% zTl(Y6%DIDP%%M3CVYLFhlSBXOVi*qyg5GsTbsGadh6u z8!X9KRsmcS9N>tTRIXR9UL-U8B7x3mYxe1y)mUMI`|;-Yjlr#u{J7Xy%0SKBNxQ&p zU;5U!M5jv)Y+Ww;o>xu|GlI3s{?}41$=&GN zz5(3cq)2|;Yn8&BNPd$f{05IOU^4a-%0%&i1JjplA%~^BhN&Mlg6V$&3!QW5Wn0P? zm|>*gfq;j<=)sOt8w&$3s_mi2^b9P`zxS2-!^ptS!o=u8A1eFkhmjl&OZLgS*qFG~ z6-ZAt*i*x^GBHP0dv7Q;zJFOftg<`PNODxOeDrqC!*$K{0Dh-=A$x=BIi^DS`E9ZY z*y42kV6)9BGQU6_t0|n3&q-A#$C~i~IhXH(O7c3wkME+Y)`i5ndKZRzHJM8uTQ$jh z0wbmO{T@nOwoe8%ysev zY`ogI(Wvp}U+$d8}c%-`h?j~V`C)O z+^KOngl2d0JF=NkVr9R244J5Ue*44Rhb?KPHvW;dVF zz#!*E!Q7tE?jOV)yX@GX4~u@xb=Vr;JMC&L0)SB8(jv@bH!m*Syz&#>u5T|Ko_}3{ zS$V41aZRDnmg}&h+SUC03eyeWpsm5AJFgf@Uvas)!S%Vm zX0R-toK8^Ic=vprHN27tW-uyT^3rjQ&7;GUzjVjC6H-v6S8n}fTYcIP*>7k3acj5F zEq;~Fj@L0g_^}kx)K+MBh>uLs6^VVM*b~+ zp&mv^MLoueW81_;FxHT}Y|%(muz|CC%qN2t(9(zqiGz#@@!1_~yhlP#8huO!m=WD^Xrdhp$mxjtpx5sGZG(;~Q}P(8h@)q`9}i+?bm& zzJ2DsB$Y7>G=@%5RwB}Pwr)X;_{qWziztGjG3Si^pkKcD6YmHvSzI9&Pp13-f!!O&@+}RBZdY>Abd#zU@fK6hv z|GmcX%#}Z5d{-`evP4nOcBX+zdMhc-0m6?*o6mp#{6Rs?p-_o&*^jNExI|s?2p*`HQZF0 zJBZE8w2MniG5d{J1^$$yZ`hl$uJ6B~cLd_S&Gr*J>y~v{ymc$WK!6MxO1}EBHsVQa z);cywx48c39BzJ4i!req5y6bc-tgeZh0obbulsiC8;4^WGrYr$pN)ClI{B$KHFgV; zfC`-Br1KQOW+k&iUqUdI@uREk=A_RFf4JQ8*MDWuH4|{X6C{w5St@O2SzPisJ9LZ7 zyQ;mZ^}ep&m!ZF`w45ZJnsAQQmJ<@H_i1J`bI+8ZS>Dt?yiZ0K=*S-M9;{1Xtec%C^F=q3cA-_Q4F}^gU`q zh8IN+To9>AJG&%3-MdGN+!sJe+3!rKDP8vow75R)+9wlvve)xk+lM7F11=J~Oa`V$ zj55;A=?5}2qOwKl@$$78f5*r{m{o-AH_CJaautlbiX+z+d3s|k*h!cfiHT4_U3g2x z%AqzqX^u0=g0YbrgPkAQPDxjOBw`12i3i-e?T72`LTv zko)i_^uB6bV;D;T`E|+vxEFnKumi~dpd*O=_c63-C2(nvDFydG?&U2iERr9~QL2vY z-hESs|GW&?BQ5AbeTU7f|Hn?Ht;XBN=_jN9`_}UazgFWD850;wC$$}p z=R|h*vBl4DO7s5t7|L#hT|5G?GnDgqM3InOyx4!z-u|~U{>aKLmRYqe!ga!KrQNTU zZQrguD<`Joe28U%|E%>Jnfu8C?rCTFre#(*bBVdXuVvQq5Y9s#Xe|Z8yBanmzw`E@ zf!~}&f*O@?^11H^68sAO^|w7%uz~p>=R#W^i#CkB<}@w1Rd3B2>BmJ92*2umUEr32 z4=3#^eWG=Eqy`-^tX#iViB5QzU4kx{;a_*(1n=$c5=L4K$rryGTX>4;vPL1C{Qjry z6nwU?I-BJ3CGux{}0=V$wkp|HWc zOVw@-fR5K+bL)m)3gvm%pb@Yvw;{NNkC$2<~&PBbi()t%I3q`8c2Z5R!VZ4FHr-E8c@H$d>XaRD!FOq>je-E6Es zIdZx2lK#1a3wRA~W+Em2bBmJ|FR8kWJh70ig9$MkBQqm2=|^~CVqzW#V^c0gVbQ<8 z4!q+fHFt8d<6>fRb#-NQea~p?V8-;0larH)nT3gkg#oyO!O{JblYtwAZZJnKXNlC#E`uFFraXML;{`aI$j(<-Jm>?7Q6Q*~J%uN6G z4Sbac+{z{AU||A08QlNJcRYXY{6B4fKZl12{P6!7%wIG8(+W)WBRmh&zsvR!{;lph zECd8UgoLnwvK!=KCbXNf7~IsrLbB$T)xl=WvnCH!7{)y}u}8@=Wc8_a0p zk>}5AeIUJilGD$t4_=aso;Pgr?V#dY@7RnZ6FW&9Y}qhvn@R&~)ISU1H;Nj%xe>`> zpR~?T6ZDQm8Nw*Q0jd)9_Y0Q4T-L$zN$=9`3(P%lz;^2B_3Nf4D#bZ|O^}VP@d7Ct zF(g`*49I@QHdqykVVS1_vW|I8T%{6&pCRwDz3?2H#)U17^R&J*sv&Ze6!OpQBA}B* zMq(x@1WbM@^l0g)nrpc|pS%ynC3k~~#=3YED7VR*=J`TBrg6a1_>H2u-H7hJs<~bb z5QLrxu*+AxYGtokVWOqb2nNp{d=WPQOEuC(PooUMAi-Yad>})V%>E$Spz%ZQpAAFl zfimX&VhyT~m4~cDS)f+2GH|p_T^J<oYnc<^%3kd+i#(SiT1RzapcL2*O%M=ZBuaeTCc4YSe#agoACy@{f%99idjm zs-e*OHL%b8z2JYgTs;v2lRsj^Wc%L{kn4K_BC@pdrAp*~+7tj$V+c=4O88e$fF5%| z=AK61vA`?z*X1Vy2yGp;uBz-mGv)_1hyh}I6a}Xx{9hXu1u)J$Hx15zz6b%gKnMvQ z9OeHHgIhD4T>5)#ej0E*zIUKTBHpf6*OMQbm7KJW4i1881vG0r`LHk~mv7vm zR(HqUd^P;NrS;+foSH6f1%5!TC;gD^!b;-XD_GVuyem96%OWWPnG;5reFlrAVWc0!3&e1Y zlfJ%wWcAu-peYYfB>FQ=)foS?`V7G8yKp;MbNtsP90h^dRcvoW^S1}nZzBlV1=X;6l4usu2{?*## z^Tb7~@-0Y??+HUB0;?|=5zo-zWRU`xm=Z8j8H>{|e|_Jd-F9UyhVPlb!FrL|dV$&N z=9g=e*BuL&^L~)q*#?QpaN-iF+x9Rf29qv6BJKzN62-4oHa{EG#RGN1;Xols1cFCP z^>KW@KGNAA2^^3Wts{UV_Y)Vn;g9C{A<3Tzm{q3BlT{)DZkOKKQnkr;Pgn6DD{JwT zGPu$i9H`ig+Wc8O?(+Gfh%=RXM24INvHZfXLpRiFO=XlzRb{jiVBg^Q!J$(Ui$&ur zLy>`)VJ_q$BLx=P{_Jq?2Y(WwVDrzibGIs$x&##!6-3s8_m_LlD^7b;d(Qp{SoF90 zZ%ys2L51Pj>T%lwLdWfeGU*$)XPY?OF7%iJc&#G9b}sl;ae}u~UlwEtc{q{#DxO>- zAOIR6Pa=Vm!~5RpONx4{3j;hRZBR0UR-QpmFtt))D z$X;VNrv%UiIPQ)i#)l9YCIh?mWqY*r&-O;552n7S0y+pL6N~28?%WzlK^{$KAGo{N zjj~y6bASKwZVUaxr{#s>%c!^SVY+<>qoCdRLE&AizAsV0^8NM{E%PR7(fg)a{*4{) zSBtaMYC@1XFR-EI16gX6fF||Fp&-WMuRcBc*@zrN$UAlL*gHj~+5zmQIYO>A(px?s z9>uScnE5uFeTNBtO{gs{N4Pb{LwWO$LxFwGj=O?RJHTLW&jyI<4@efZ$t#oS)Pe^G zg1(Tgp*f~(B!Oj{*c1-x<4w=>sF*)HOa3AS_#A`zvh(QHGr$zshd^eo^mR>UnMx>R6TQK!S(u63#>acs|1P`)4DQXcX>e zQjzeSPrq80M&+04EY5z~c3LzF)gY-Pc1(Sh3ekBx-irgA;CI1k*udclwE&H?5e1Gv zX1?>*h|nD`NlRJW#~#tUK3uFv=69s=o#H5xx>>K?Qx0|U$dmpJpQHHfh2EZIkMB^7gelk zE{Y!sD;|jNHj_>wpeOn%n5`tqH6Za)5Ke--3`LZlRTa8gKN}vg}%2f#uA$? z#g3{bw+X8t1WBvW#6OSeX}o-|f>@Glmd7KKGz>$dsPejVFdj~X1N5o@AQ%DE0qmca zC*X515S3Mx6drrAP5AjCL?dxfC8MOnX#j~b3ja_k1e^y}P~M~a2L!F|D$SleyLC_Y z3ejBUA6V`wte8Bsrs1a|A_B-ZmbXHf>gCI%wu5!fZI5r)35Ome3qSwji+F6uncn`` zyJGjzYOW0BAnWk->G|QLo^+wv(JY^odb%g?mO-}-rzZq?K76&y|JB)|S++n~AjSdM z`tn}4{bS)5SRupc7$%L$H^j`ta}N1aoc#^Y@IIK!qx79@kEEcUOIghOSvQ?6LtFO* z_7wg(ZBB5Z(55fXPaeNij1N4{x8P(mx%vpY?@UIs)gfOVFXRZ0M{`~e!{pbR zjT>J6p5%fb1vKepFj@6I0gs#f15)Cs`2QV@dIo=|&r_BCrYMe!zUWrK2Aw=9yZE6# z5INMDPm+@M;Zw%w)-EX9n8O*#5hH@9ac4`20v z1?%J{V2`g;V=Qs_t93q3eo`4A8P z`AWS5I~}mbuk3sd&lL(06!en7KppmlGCs;Jqlp@B>3gvZ;}yHwpY55<7rWwp4`^}X z@SICU3_ds^^-@)aL|PR|vE(>szjHf*YOrfEXEaU+c7FIHQN-g&KR5iAQ0G{zr!i>W z@;;a=zrTNeeqJI7k=}kf?Ll73PH(!7MRBOdj9N>)W}?VdONgK=b6T=Zu{4UW6A1mC5mOkmBv2ayoq zXtkeN`~5;e5s}v|RV|;IGqZn|w#q(2Bm&F{9(qG4*j~{~f*pgU4XQXi?7hr!*7BuR&JDw|vA-6IU^GqTfj+Z&%IKn2#>VFzWEK&k9_9R)md0FT)=%a2( z@u|v;mH~!k6ysaxVOu(!?IelSFc8*5vu2LSjF7+&^_oY|7iwJXaNyuQ`z#S53ZEQT zlUF>XIMJEB9ct5?TEIK{k_u>v_peNx!D6w70tH%KU$6S?R=uYld)%QqSM9cc^?M(k z{9yXE-A14~3jZTNh)uAIZTG|)3MpOSk7tU)0k*4uuFgUC@Fp_<^xchoN=!8f4T zfNENKnVBcPF@DkhC>aiD)l-sf+4H7?>qwVun{v?EOOZ}j@y!BdgPrf^KILOi57*Zb zvRl(75TaRcrTz?e_O)9R#oU=n-)tN!g>GD=-J&baG9@|v?q?3T&ClrrgmU2*y9@ge zORC3}6UPV&&jsEX7bw7l0SA&I4eHZMTbS`seD~}uYZ_}c2VCzb5DjmFfbLwfA0dtX z%lV0$=Q#UJVN=1mpV)(CZkAEr5n5H!!-xx9Uskd`qWznw}Ea zRV?HR2Ir&eqo2j{xgg8AvamMyi;a!Gh})ya)mTcIG?dXYz0Qx%ce_MCzPx({KEUB? z1riB@TSg$T&#fNl2R8XSTZryOv3~7nzz$TBnVB6O&y=9M9506f9UDE~o{0l%*Tft9 zOIIeHt#7&6aeZeecr2596VMs!$?FneT@nfh@}Zrp9yk07@y!(;Zh;OGh==FnYHM90 zL@G6EP1UOnp_3VP!k?cW@^xBWLW%f%ig`Jm_Y-xI0l(IPSBGAyXCQ`1Z2kLZq0w<+ zv?>u$$gI#v&=9))&f!~fi4A|vS*skY0ohaUb0}|^YvlT&p&8TScYKlGI|5;ueKu@4 zzRnH#TuzD=t2V=T9L>D7-|R;O;mCV!=Jen@wA}mq&ydf2#8I+9HrdMbE+2QNRmjbU_yS{xD z^;#bSm>WF5NXzE&klxi*qRbZG2tPlFNIYH`LWy{<;@9J1{(yT)&2JPph>ilAl?O4F z#fzs@DBTUL*E`c6IB#&6A44G4AMU zq2Sa*y7ud>ChAlWj+z?8y9Jx<&Ok2iC;f8-Ki{3D=)Ln?S(e93*3}rmo3#xMpDiUV zDWdwORvzp-(-T7-NhO#4{m^1MFwyh&cU;BDO_*&RMpxR^!Ny)~QIo@tSL}WHox$L# z0Y_z%jk4t0E`#RxzR!Fb@7@u5A|E1OJkwuEgo)Gn4Me}>9~>+kr@yB0vtlm`q0S}E z#$X6`(QBL>cpPtox}7)H%_=*31=YJTU#`V&^OVkEPYQ#C9}3vG&DN8S9>AV5`#yUq zS7^U|xH%E4HthQ-k0+PlXM+YrGSEv-kD`1FZnnStL3if@Ww_3`?7chXT6TBmCQv-S zU-4FtT6J%2hRNx*s$O3m%*|S~-(Pj7v)i5y^bQs4)Fyvq)GKE}!sof|rY1IY#t@O9 z4@+;bo_l6JDCT;T+(XdnvT=ClC|-uYuS%^(Xn4k5K!3=j+u@g;ovo@EQA(t9r}e8a zQAMF^8LfFeVWTgOOv|Os>kcg><}r#swRUZ|zBD$3t&&HVI(H0jYbt4Eq1g%>IAI~f zm~`u83H2Smr6DawFZe>|TSN6C%I86phLA~CzGY?EjWf80ds5}NtU@EhAKs&X`MB*m zcrY|gC4R49xiyr8M}_p#2(>$tI?ds6_=Rv4t&dLtX1iZX_&EEB+m)YClDu5HWGYv* zjML{F%FEk3j>-0;n@7;{yN`~?_~hcT5N8|xJ;2mTAky5u5N8!2Kg6ziMxvpFj05Ie z6o>$s4ZcHN?$1hT-5Y1>?@fI*s$*!wf%}r~_xSpwV(Wb8{Ubv8qr_9*?;rP@ayD=n z+cZ2wdDBu#;-kC2Vq*TD{fWL?U-J8*NJ&+<^#ret4sHcbs_I<$d9^f0nnW z(PKdv3f)*acjIKWmfGP;4S_c-VpnJ5r@hDWPK%Jt^oBgGlPvYpW!5n0o&HYtdb9@n zS=<8IOk7MWWufDYW;-EHJKyaa$rP=6w-VKfou_&TJy&z&o8GWl3|-=tHt$!QYSmuQ zZ*F~bP#0NiAS;M3H`YAt?4#YIu zBy^2xz3c}=zf4NzCl)!F3|p3JxiQ#Ym{`@>PGhB1>iVm{Lhw<1lj>kHFoikNLublE z#rGYy{bp@!_CP#^jbW}#t90CEx$W(cN^d6Tw3*?HtgY94Icyd!-S{>NRjhf|WXd`ARgumkS zjxIah&)^xexB?^h-s|5VYH)m;gN!(5cUsfe=JkZk!x3L9C{c=X#zpEKZZckE!BDaC z=?hR-)JUDPSx;DM}`jJT@wOsU{#@?=V8(hg{}T{TQlT!^OQt1zs#S!$|Q z-8U%_+*d9Mh+O!;D=9v#w{-AmFRXa{hjL1^M>t^f{883 z<()y&l|JIadn3sdbZC^)p@+i`=h`%;qy)2!kAS`-1&#)_Qc1wG{n$YgyRb9M_QPjQ zN7tQ0&UUP7i;*3!7>mdCDGI4Qi)|2Xd`u{{Z_!xBD?VH>2k>r`JCj7(w`fxx8ZeStbI^*X%q{)4(2PX?S@FFt$CsiKn{A( zT&86WrtFq0Z=yb0ebYk0xkb8bYa>?ews>DiCejkaET?h2|LuYQ7&7qjblSXpdA3+n zr80`OAiX1Edwy~8?G4fLysDYT7_nq-8ClVn6u`H7GM-q3y=Z&~HBD*pIgNt9>IW^7XO|+D%TVJ;7cveHU0?vgQVYSwC;o z3=s$GJh2>E$Cmg$Ln_>BC$Z2@eHEhOEvHjzHFU>kGp`~kJ<}=l6j3eL$agPSjtdaB z?n6uLqT}UBPxf!)jvsWgErH*zHu@sG|xB7POExkQzb((_DrkYaVlV-6b(h8omFVq0nZ{JfV=}6vs8#9+{Srld z=Qw5^nnyWYKp)A3CZZ;uSDf@s9qB5aO*CuSotUqml%Le1dY;CY)r%7X-wR~I@OR!p!WD*IUxH0_HQarI7jN1n#_#6*c@;KJ)aXlV2JRixN zXw7`r(^G?A-px$Lss6QEg5~(v`*S+B=JA(imS)WY#uhybL}EhPNX{p}Mq;hKQV@wq zBX-8-rBIDIzfVCUI4@btzOq8vd5nS2;k2l!V1^si?6lgC;H4$y9y$K@xqAa8O%QY~ z$nSL*g~OuE5Z@Vsn(q@tZ_1?CfX`N`QHkA%cTZBG+smF-dfd@D^{29{}R_-k%i8{n?VrO2g{ew9tOP;!4OOwagss=ZNph!Vs8id3n~ zgidY7>LB=%sJM1m6t##$*8FG5s~WTMiuskNZ;vR2qiGU1-D{mPY_)^cA?H@qq*$VN z8Kmy;yv~gdLyAutEINNv$s<3B=}_KCAAZ@y6&C%-u}*A$+<8o9e-!O zaC5)R)G5QCUa5J&T3J|bvvR037@X6Luw#_zaASy~anr1R7n(${iFhzOZ^W>vQ|UsG z+2Am0*-p0@n|(4#g(7D0`y@{f(TW+?-_J7!b?xTaVj2HRqb7QUiHIy^sT*4%mB+ni z@+rdl{xuUU7bM&~1uC9#i!43c9NOX{^>lvTp@^vQuLN}+<9V-P3>s3U56f*e zS@LCJL=IJSpZI$*L&_*3%XqHO6s(|h>RMCROt$Z91pExMp?Vo&=pkPpWhKH!4#arJ zekdtgf+MUxv3e*#aV&Rnh^j2#+36r<@Y|srz89n*=JuMe(1{4Nj1n{5-pB)-)27O~ zgdmD|wG6&S%UMb8pk#4kRF&uN-xBu7^%+1l59~F=*>$t}ItWK0Y+T-`+Xq6!*)vSs z{`q;25*mAzx43U)iXT6V_W4X1H@d0c6i|B`Hs_z4J#0!qPh1aB&(&3ZTe@>rFMq26 ztWtyRY6WI@hxfxqoCB_9?5L#&;IdNxRBuKD0Nyth+RXvkKKBy5GE07};}hZ?ewf=3{a3rT%&rD5N^7+(Alk!mH>Eca1=aMI0Flq_L1ibiEFGy!UrkAfNOTwZNq_H&g#XJ|KE#_&r@Ac~kDl?P zaNPT(O76OOtnN2^(?Wud5>2g$bZSNi*EU?kVZ!$dRTgBuZ|$yVRY;=!DDuSe?-8X- z*TJYt6w2#+>`!D-hVTDM2L+&UlyAUNDILdt+|rW~92jb8Pl`zXZKs(Z9Mc;hf>77>G20^Si`7lkH-Hg10ZMeP{x`nY$`bP9#ifbt5g-5Zz^3f zx%1*`mcz=_rW<4}6Z-d^trVJ75!<}3(9|mRrmY@m6X_~fT~E&D-1|W4r5-Zk`132T z?hR5TQ)mnVXOxhZOnL@9SnKJX!%I`bS-)Ap9tZQ(yk8`4R4G%Bv(~7=jW?Y24B&a| zWNq;`_5nSiD;9kOq-*S`LEAHh5uI2}Q=c@9$%1>eFyo@IV8XE337@}kyzPAaAo}n+ z&u@(C+0@J^jim_0;A#~&V@b3NP@B4DyT6TkTa_Q$y*E-7; zP~-8ksudDCS+<()U`wa&(|wND>~O3MIo}(CXxb!NqwW3K7KREqblfz5-x;SG@gcO6 zqJ$_aAlXE&{~l_in5xs2Z5kKWR=lfy;Pt7e5jI9e+qr-XD<+0lQ4Yh0kIc*{;Jn>V zP}ZDnbS;8554uJt0n!o(ZhpIV0WV8t#5^L3*C;j$cdy1o^yTDUUw_pHbw9kxS%N^4 zqpxb}t70{%rTAfvISKz!a6O52t6aW3d`adJKjh|+o%}2lP)OlW8QPmdyd@&lP78hJ zkZ;lv$Ag5resOlF)>xn23DDz8Y&@Gqy)MGz5dpaf@w-4YK0iwR$l%6%>NqGbGUl7P zok--PL04_bc$i?^JRO$2bHpAq8h`e_wYt{669=?v6*%p0pm_Jf7wZy0 zerkt7uMBe@h;$A1mS)#Yb~(X-AaL2?02@hQTAV+p09Q&`b8;f2*t z@NXxD30k(25GRpyQkz_!;u{qN9);WjVKmd{*9=^ZLfeH`abJxMyrzp_H_W7?5Y!kC zF8HKziuj1QozYBi=afyf_MNh+-J__FfrGVrcy&a3OnSw>j6j(BejcC6gI_i|3%@{<^$wHog?@2qdUljUZ9A}EvHR!7I_!a-iHy2dz zU9DIxOAh+)-}1&9R$$i$;M<%XU=Qg*n7I&i*^-In8;mYB`CL$nK5}8zYqIrR`Pv%3 z#P!@#t5_Z`HNqB+Wr_@M(NsBXav4=leXEkUGG*$5_zE$c){SBLzCVot?`r0FGJ$HM zNma*4mopYcO|e5c&d_|em`3v3P1HroT1>W2_u2a*d(pnPGD)Nq4TnqqrR)`CB_Sch zogc*s&IRV^hn>%N$MP`;So$P?x9m>A;{;%$|7CpfV<6c5Iaa=iH4W4R%NE*9Q$yP> zxIRr(^kKU_Z$Q(2sE7z0h+aGFLug?%Gcz;F)MK^!#FHGDp;2o=mw_+UAu1X6X!MLP zt;eB9XDnkmpP(e_j(4gmw?`@as7FnsKx+bjvvc&5D|_#X%WFmRh)`Ca-j+kX%>(*T zw=-Cp$&5*7Cbq1|KjiXv6o>r=&NNuj{<)&Tw72{yH}ZSB@VQ*y=#BU^w$V&pSWC5v zU@LapwY>|oP$DM6a4hN|42gvG`WLIq$HH3G%iw6ds1KBkhSEu_%}xnih|JE%^pTn8 z=~<%J!>M%Q5ynOKxovk^MBv zVq0U$T)+|rlt~g}Ui=5Ab?kKPS}#sVz2xqxo6nBCtfM2Wv@}OVx(5;^(rUgT zDw4=&H-pN4X5GqU_WH2zbw=Zm%urG8S##jFQ;8l&Ud`Q){Ggyg;rR`bO4q}3zOV1C z7FGo_#_S2zc&5KJ(?|=SQbfTm2m{(~Y)V8KA`oR0_AezMMus9*-t4h!p4l+_472PR z3DL>UHqT?9OBz1?6oOn2UEx$M7t1y(*6%Vb`|}suHVf9zFo| z8`MgZXsJLjOl!fGB*EvXz@`De&IXnuE;_>Ng{%??Y7;?&I$dm>%hvzHe^QSIAfAzG zlTkqC!ui))Z^nAzdfvUZYa=kqD4&&D(c2*-+Lj@fW5@5l$fQc#WV>Lcq_hlAXXK3c zReEK=A=-TKZceV#_Sln>H8G(F`$-hc)cIDD1YC8d}n*Rof2{t5iF!2-Y`hJzn0%GJhS>zkX9f z<$Zv*R?G_dyAkjudtb7thRyl>kY?>&*i?ZuDo$x|Lgt@v4Xx?}B#^&`ok%YHanB&* zr__~9`Qsi;1pAvD)(>_esv!*=|84-9yQTnMnPwXE0o8v-9}nz0_rnB@{?+Rrb0>vf zRU$(y`w4D6eg<)j@DuI#LGHHZlL(0H-JI2rQmZg!*-+19r)%A>np#`Qd>(IKD9(ZG zd6#(C$))pcW=pOsq~rTgR0QP1NWWRkF1NQz-Q`}oO-aLuaMM`YK+ZN$ZJ=_aucOwj z;fQ0UJr{jW6^-NHPNsj~g=*J_m&T&ci7CYL4w{hjtd(%bL0O>;9W9LWeWRV;ZkJs%4<3%90UBcO0040f=#J=w_PH{&&ow@K8<1p%4nZdB|E5w}Wj0Ru!F+;W%aaKR)&TC>+HSV*+$OL6hVbpf zgH|zX8CwH-Y)I(Y=0H9e4e_|Lv_7fOxn}CeGy448By}*lp;Jj*h`}b|!##B_zVkW2ef2oe!^{H`N?hW>YU;?o+e4IkY6?b~+-t ze9VjP8uIj6%=c6ONGH=iB1xe&;rLJgq66e~$}@n9NE`sw;(QPcFhuv6w)kEim0kuz z)RaLzzSoOpnveiZuv!%jiI4|J7d5Mv1O%|>g4%@l_&hF)eUK~NYwpD8Kj+=HwzTBA zU+kP-Un5n%9*^_WWh2JoQ+PCn8!FSNt#Uol1v1k@Omy}xE`_A#MmG|XG=4SRWG0ot zB#E6}yOpPQ8=Bfy?^HYaemP-O%C@auuD3cNAHT_nz_X!T>w9*@%{-ZEKUIJy27+p0 zOQ>jk&b*>0V|e+(S5$IY_{YktU!_t`?{>2fuKeB4w1aKp;NWa!?1iv560RRlKsUWNuc(3l z#F7=B-j6=Cq!MCHoePF@BKdRfwXBxvO0xFRrN(boePaLt5i0YyU43pGd0wiwy1hTB zjH6RyTyAkj07z>60ADng#YFOI7x#MAs#mjH`<)1Xp`TWrdG0_fK)TaRWm2!oX#k}+ z2qf|bLIVUIIFFFi&j{GWlAU|r2AaVGdAD*5EKQXdlsK_DrI>#_D=sU|Bc znRqp>KaK)KpPXWzoCDOJ@mj4~(=Riam1Q4%a^5y%CgjpReRTUul{Yno6YZh~3Ce^3 zh$_K0c9^v!mVZ>qF0W1{*`Dj?nOMmJZR&lT(8q-iFFXn*Rt(=Ql;~)k(GCO zoi6mrmvseyR=6eF58KDi;|Z&Bwr5l6ijc0rtKZyN=#gufV?-2L15drv$D3RtUYHMJ zkP^k6IVVVtKN7c_fFCbIC`3guv472kys|tgB0-SmNir^?xL4uRqOoW$fG4Wg7{dX3 zSy(<$b0$td&dA89$(fiM3=ku7t>!BKZtc z%v^2`#BBCOV0#1;e-M#RP6=(zT?=3R)#_S5?d}eds~+OlhuL+Ts`#B#LVo-lZBj_E z9U7Zz{#Jac1z!8YtpBH&+?m^$%WY;eDi2;SVq}N|;gk6iB>^SlRqT#i%MP(u$_YYT z3LK#B=cv{)7*hdn;&g)*T)H7JRww?2DM4jP|2U%KgX`zAaeAoBx+=xl zt?g{Fu45d7Xlvu7T|Z5fR7AZIi8T#kW56dD!XdIQUv3v)(W@zx!{?Pt`yf$C$EpHl zqPalqdc55Idomwj>x~VV@aCw)(#( zJFdV7jzdoF%^B(!k5eQ^g#U!Jf5S0LmNJ}T6Dxk(`e1HID$O(g(4Vq7wAABq_*fciv?41tPz_KJp^o#GR7 z&x=UV7o(jLoz@1`46eLenM^JfIp_!x`R|B| zWnx_$h0Kd3c-m;i^`5yyPxf?9iO8Lf>}SglA3j|vM_UNdyevPw(Isrj(R*em?0VVf zZM)xnP}htlCq~r4GfURrcG$gsnuw6vXlnn=!;W|0Z3H<H+^eJ zcWmTcpAXz1MwR+?FUxeJt=e^YQcO;fYv;ERs8#gOENoZqunFEy} zmi}-8syE@KDBC9-;Hd9Wxw&)M0$C|?y9dCHatdV*fx#HzVw!fUp;N-}4qKkTzx7747tEMC($Qo>HGz7l}yW1aeT|ePmR&S%N+%8 z@9rK-mg!vimKCqawpHh(W*vu%rSBP6Bf5&1&a8j(5iySABZ&~BoU8?LAkFUnc9{dn z3qZZZeC6-CzVimTE);|`RDNsUvIRl3Dk!i+@pV1M?2`g2PmQ^+Z*X)IxvSWc-p}u5 z&y3sCClDaLJ3Luv_n7P>GIA>|8P9V$b>{bV<9sFxUilgPe2Q0!hFM)b?3JgN<=HHU z!5dRlJL%{f;bXK=2EkB-%8){_27kDWfI|iGt`MT5`5@4fk5M3PodNP{(>j*pY=d-9 zEoptS;Dhe7H>!`B?f!&}gYFzWrLw^~d1QjXaw&E{H=i$0MoV{WuPYsjTOLLiKBq%* z?xQVz`9vf$6U{-|Wz8!S&Oc1tKL{!m1>pAOXkTx~Nj1HU&=%Y_F!t_MmvA&wOmy5y zKS88rvGZaP`Pm-dt_^*ARX7ADh%|hAgoCM;9$sRD|EJmsI1cYVb+|Wd;~Zb6M?uuP zFnuHa2+FoG9jq%pH~I%fC59ZJM9IhEf3>ehRk?1O4tHv;KzLjq9$W>3dk*cJQcSc# zb$S<#-Ys|b{-2(~lyQJQC<2+-5C(m^o_KYt)1Y>AJQ@LnRQv%vdqNzcNT=Sp^DrGe zZiM4ESt4KF1(GconEnGh1K5oil|C!<1jO^v6V<6si&!W5D20Y0hiA(kK{uqFF6pWJ z0!hRh=TiW8|2KLj-he8rfPk^x|0y<3%=GUx!Te>D*S)utC?igR_3?h2XtBS>{Qdpe z;P{vNel6?};A#=tE_%P5Z%aityNB?ITGjp)@wa1IkAcirj;qm{}VLi=3xf zZ`se|5c|PG@e!y9XmZ#NZ+P8cJ_!OsfPTO-1#bNV%K=G>V)W2>5Dry^V^B|{NV`Qc z-CHUFo)ML?G$D!!z?qN4WrYWV^?_=m0r^&!Bl(NBXi5d*{=i#sjaHNA4I{Yxp*%47 z!s}mPB$Uga`Un&a6x1fR<8A<_L!q-KNm8-^bv|#g%*lD6{SYYek_QN`(qEVbD(~>v ztZ~AznFuLS&M4S`I2)+0g8>3!(LCX>D$kowKwS*Y4;G#?15l-2XaBtnp8fd@4Y^b@ z#?`@0&j68kP|A@TS|O?FKj%`=bOfgiWCW(?JqRiuZjEXMIN)3O@(aYqiO80?4MSI&y9yB z%IuYN*xX1sI~{Krw|>N-_l0A?`7AN=F=;gmuG$tx0iocl;Uq?*v|b|wRF1z`nnLBg zdMjS1m#43|?8awT}sn;V$5nvYBZlQ?~4v%(|avg8>?ApCDRMAjq>?Byl%gFYZ>f!kG-Ay|CoV&w(argswG;Ku7%lqN{l@-~llvP)hsw806Rl!$j~v!P>tw48!Z~W49kCDW9~PS|5x*&a!(S?4 zo(5>1`9wj-+U=0;U2sxcxzu%?5r-PJ2CHMg;#7HhT9Bq!(L<*&nAhX@mEphvCYUl0 zhz&7(R@q4cB5@6gFb1Cn8MSJsKp@OI;1rkxM3yGGNClga{3P`4W`SwSdokx9$oY16 zmz-s~HE+O~pm~BNE2}ktZAn}SzddZlgYH9E#3x6{hsqc8qLPw%HIV&U{;c`N`ezux z^u8pVSe=v?E$%c_csh(wef|1PSIagXYppL5;oPwH3FK2On~f$f3=2{)(X>iL%6^^5 z1=d<4O+p>Cb>Mw<5)Ty;k$*%4g6Ip+b>KX9#2S!wKii+Bb36GNkq(pq6#RICMXnm# z9k_ykU`7G|Er1bEcG(Or3u){=UHa>kZLiV06Q{xVD$#PPMnTs3mIW;YrVPM?ddz z4GGvVIUCJhI*8|)Gkgy>^A9i%euwnF3^|S-pJqMTxw!P7C(-dK2knmX>mSDCU#@oW z*o9&+X)60BpM;QpUe!w7{&?YiPTo7a=n&8r#$@l&Fjs5zRi1ktpCznc% zH8r4FgF&AqlBr^;wqG;>|E{CntKyB!SQUS&S(1RPTQ4^#*QpbS5-P} zikZu&F&VA1hk~IUOT)5nW?z?nJ)cgJ)jjQ7@n+Ixj-(GDy!XuxhHQVZ9Lb9GZ`0O6 z-|r}r%@p;O&7@7zJtq!(zx$&g*?i$?rPbGC|O)G|#hRfL1) zqbaN;2`>m$=l?2O&1N>7MDP^yW@?YawyI(yT0r zs8wWy6neq)I*ZF&iU$r>Nj2IS&zln=Q(IhCTbP&`nyV)gcjG%DyG2iZK8LLf!2{c+ zw?wo7yJx53c`sO7wv?6kv{FxoZv(Cvjx_=m~j208l$oq2J5G@oAX@D6xA5 zm#7kMs0$j9>u9`&(i56(5)k_F zi;skzRCYkF&YO68w@|S}P~^=q9|UKO3A+rE{c_{3D_6{;Uoy2Gj0B%#lusb6=z_#O zTz$Xe@nz~lvwLYnhRo9ivUimI=D@+bhnp!CYt`X`vJ)P+-zZ~*-2T1$O~o~X5q)O> z2Y;c)hR(zJAX+P>(qM{d^J)rm+BpK5=;K84kigTuP+2%o#t|5!$v(G#Fn2%j(EoWv zz1bWtx|n7k;5hdB)E{)3t*-o@YeqQMdBJAMq{A95|JnHYcP9ynE*d*WE5!_a;18V3 zSr)q6N2pSRdsS-cLH zulGM#`vJCV-&3-zC3gs5RM(-=Ami3t3J_|KB+_GQsuqd_`bm8l1`6I{{W4LQy(T5C zSBk~K?Dy;J1obv2xQ3E?0j&m`9%R0#{%@>Sr#CB4b#pL*Y0TZ8rCI7C(jsNnKy~Px z(s7>MXNtE|Mf1Z+Wj$ysrSlbiV3LxWOlcOU$73L}#AgHy;k%k`r>nP9U+3o&k++AF z`&tkWu41fG6~9j2r6cvzm{UzPX6%;i((;j(mV02iyNqi7hs!|cKN+mnDN#!owYf0~mUUST0@cqV@>|`A3i4S1SY${?b zuGP6MDk{t0IJQ>)gc==`nPhQ^KbtnyU{a#*Zn8I5X7XHn0pc=`|iWj400>53>yhN(Xsc{0{AO&Y@mB}8a3?_~%=Qa*(#!HNTS zb~R!VcZ2x0d8Zl*k+0zUvH{NF=z4MJ->{xR$2W)n< z>!)h$^;job>dDOQ$vpH#K_hb0F2DNLrm8n}mOar~NjdqUl0A}H zW#ppD7ss+}y>se-go3+(y~bVGXDrPYLGL9UU9B#O?RbJ~nO62AG&BBN$3F7Z4L zMKZZOArB?HRz^>`0)dH;P5js_!)QBvZB%^_KuaKi?l8 zYdx*z!FljP%nvV}F^{tNXm^fNo>K}xosmwYHjzaq*ByTNi+A^$cY}nT2D^3rnG&Ji zDQ@3Qw(dYJ^au744)%sUK(GelpXJoLn zQ3@RH6qU^*r{8&L3evG*6EVF~$j{!8kTjWB@k=#*vRR!lJpC)m99B;e4mUC-l{>oDc{jbIJ0ON?Et*`RWPsCZO`9@B5Nuaw8yW$4=MltB1UXkW3^ytvrn zGCj+?YnFHxwArHQR!AaoO)szdIWdS55Y=eE*fI<3Hm;1zt1XEeY`1`w;=K1JVDOk7<8vi_o`0cxE`|&lnI_vmdJZ=Ju7YFKQ_x+Ec*xC>{D2t`m zCOsx2Exv*j;zzS=IG5j4Xm~2mvEq}uKqm2d^8+nWkPi63T&{GL#nt9l8_`)~WE3u}-}# zk*z9Md`++WKOn%v(R|aD@zMy@6ndcz5ds-?%DQcaMse*qF-kjSk=Jogd&ECB>NjOG zI}Iizs1AkWJl4rAom$=7E64YA9HmTrp;~M{NSJo<|!LY&C zz#y}pr`_{djmfq;Ydy&6K`KocN^AU`yyF7ZY;=)+6<#>Cqh>#S%lQWJtN3@84v(0X z>x{VFT;*~rC`GT4pEZ7tu@J4U=i-X79&EE>G~$okAy%98@*s7D&;EW>#$)#~RDLdo z$!UAYba$acKeCL;L^-Wv!zDA%>eY}zI^%N^5?B@KP(yj&5()Hc8qG^=|ypH(Dp#GREqc% zMbu>A5tsGM*Yo#eI*`^PzuX?w6W|&Zvn5d5#5H`7rBn*xq~cX}N6gXD40fM>T>|i{ zMv8wPUgsUzF!Lh9%s$xV+My&6IXs-LG!(D3T_CuaQeFi6tbq#JQWQ%MtsD>43S&j$-^5kd->rNw5rUJdXnAr3Pxn`A#8BE~|+{nHTc zL*e)i1~&imNu=!5?Mz%9(xQGzk$~oX3fApA>WClMNl^I^A)5ZnW$tWT>LOPr6?zs` z&b@Qa9(IO_WjX;KRu__KT6L!*!<)tObDkuVRIEWhz8RTs_YZD5dRIF3%@T^*kek|ODzSD|ieHXJ7TDS`<~IX7PC zRM8vgBcthJl^cWaT`RsnVSL*#thkz}y!4&|&z3ZpY2_CnRdd2cX|(=SCdP_8%YaNM zf@8F|C3+`zlya$irHDWi`=QYWTi-W$%w;V|oyV5whjZve61@`DZ=RPrHKDW<-@eJ| z%j^*H@~&~@kKOt9yh>?22ITOX*RMt=Tq?|L+P((9Qx+YEY!g_rQzIE}7Ea3(&chQ1 z20f4$%XWTQ$l{o5%~+&dFka=wX*Q6#+)KES!D&t*myc1X(Z+a$&l2IJ%4nH4Qmb=d ztaM59BIIWU08&}mPD8maOBs&v7&C7UUJ25;=%z`Xp7Mw7QfAN!^iP^AD1nNmpDb@) z&L%LZyX^{B>Tg=#N-`~`OARHvX5sX@x?qPXFk|}P6Y9?a>17*%pnYGDF9qM9fM5|r zI#)w7`7;x&540e-w571hv53z>DY)nh-#(~Un%wGZpnI^1=6)4uxBkoTPAz0JM?(*0eK^SB^vh9~|E(g(W zmfvEt0CE+nzM7w)^>kqobfLz*BAlw`;l=T<$<^<$GQ<}>v^@Y`cX@)7C!QTKP+J{p zqoa8_OSP_2MfrLFtWzVDUb{%$0$A1nAJ}KvyEH;tK*a=9qVfsyL*sCs_p87PerBOW z6DICup_WO^goNfVH5v{iMg#@#cp)^a%@Q=ac;ZT|lCSVSEc!3`uIA)NY_e&EDx1q2 zJ9Izow{B1xe;Uz-xxbN)~Pq#AqX(gbc4LKhxu=FlAmGQ~5I>pkScB@B=ihd?#|q5ZT93Tn;}*zM{q84+ z9(S#uz1Ei)zvlaPmDD4Edk1z)D`7fRYT~2l_bMG*h~fAw`DI>4j_3+bB~QAH{T}BP zSS%{jN^fX!NAoI>fg16rtwznVB5Z6bCE6rm%1^#To#Af1~e{1Xe(tO*&Yg*NfY_aO*>L7b8*;H#Bz`r-w*tt^b2 z775_9O6wDZ@&>Xzf`|MeqwZweijOLbZ8{o~X4}Dp4lkwrP9!fzCuF5?{_;mzX~qUU z73L!cisc-Bez;;i3LZ7G_uyech%z(XMSH(n>6=+<){ja0PGhv^4PCjW$vk&s=t9CV z;RusEiygg6roR#@?kAQbUdD~RA8jSP7LQ9m9Jy0{76hP5InU>xil6!35S1ikAUUE? zPu9hAPgQHIIIs4*&v9n}vmx=t8`^Z4;{b?kAbX~Q z{<TV#C7Oy<^(17(G2U#$#fQ9)Xsj+H5! zPg(v@RM;jM_{8?3Jv7#=xRZ#3B;R<$Ic2Ko;ozk%UN_ZyG^AIYKG7YNj@-tNG>t7T z=eXJ|R=)&C_5E3~#SWP%N7Ws_{J<9Jc@N&}!YqAw*{&j8sGRAiB^?faNz zgwD0|yH=-C=+s$D^_rQ3aERrx(?LB&B|ESS0_3kd_abA(h$eEnHqkq(CB4CRl~^>7 zN*)BYYw@bN#{7$={ga?&hoR*0$Wi}`M}q(!&Bgz@8R7pr1{2RyCKiI}O8xsmq3D#O zz+@j7|DAnC=yrst*X|W&lKv`&PI_cZjVSvXV81(!*$lC?SdOfWs90p86MmDj!}+S% z4`jv7lGE2gWz2nEgAWJv`Xkr8(iXJ08bkuQm$kcdS0S&)vLe!jEa>Asqc8fHW zt!p>$L$=yRvBvFY>XwQr^VRRnc8z+w$io`amn)Z)!d^Mr3z*L*E&~3|4v!X4`+9I)V0BHE4(f_uu~ZsDW2w3>`=*(s$omo}SG2bl#9Ex0!oZsxFGL zGlMRP)*3~za&>|#aPm8h)HFOxETKAy!#lYB2r%MG8JC1;`uG))!g!tnW9?_STLmYoIB}PmTc}V|E47YMe zyTuWMSJXLKua%mVJpOr`PuLraHwBQ_giPfXW4Vj!eD2#+mrODX+sB(@wlCi{HZ^6v zDdPCK@H>&wn#!38p;Z zK2#T0_0?ziC@oLg-xDLcEpYn&ocyny&_lG~p17<~&;Pbx@Cj7q+|1EjM1FmE)HT&q zqgZ4q9HIDnPP9P5s<*6OtA&n4JY}W~3o9?;hX1a^)u%`~zo+V33%lzHA6L1qg7)%^ z%@U%fw#V%eZJz85`b2!eXols60<(K}I}|1P%Z+W11s*`!mr(vdt$9i{QTS_2#D|0B z;^B#~-rjQazI>A98-4l~b*;Z&A&ze4wi_kO)bPGs4j47iI0Zd5E2LglQtDH)ozG-( zcYZ26j=%e(0mbN)gmZ%z@5GBdbLnaYqSNfAnlbuG5bF)Cf?=yhQFbpjo1M-L|DrT` zj>KxVMvW^jN8V^=LRVki4)O6&SGRN&sIv0Mu$&~C?6@t>6Co(c9QdWX;J7=5>Z_w; z`|?#BM}i3E?~eAgmghWWq1QsLIG0hxAA3vGcNkX3jyUg&mqGT76sU!WNk878KU10+ za#ev|#WK#aq0HJzS>;*a59&YK3&ZW+3Yy6qh5Fx;m+d8{V3qRIXWxnZRh_t@z0% z$aWZ6RJ3a;k=||Yds%5^JlF&)V#i-=o@s&Q245h(%Krqon6T(b9K(4^Aek?Q1evLU z0bzP%F4gn{wQ6|0M=p>1Krn_Q(4;&p#hA9Zf|7?ieGs59{B+d&L5AjlSfYpft3$?2LD1$r$mxZJE+q#ul%E zO=h;d54Uc8c5(5EYl(_9e;tv8gk44+RL!_w&)$^CW!4%=QVEyV@w&EX-iRK0AuBn9 zEu!+7W1h?2w3c8f<%>;rYEfMnr`GuF*$77z9kLMn024;he zg!O-N!=2Ayo-D%AjO~AUvX}oa&+2r%M{R-Vk|0cy6F}0&0D<{9>3J59_!16i9OK?U zVgziDt1}i*MqJcA_>h$E5uiz>fC!}`kim>+QpW_TQ6~!GT}D(sR+#S;Jlh5Gvi=Cz z(FZmFJCWY|(F7wSYwZ?^)ys9d0L=0Xs8*(mz9j<*;OBsTR06V(yUQKR0b7+NG%Frp z^(nBFRMQU(9aG8xsJ9&)9lhxP_yi`v2k5NQw;<8Q6FU&kfNS_ghI7g1*Umf zj|Rj#Cz3)?K+CESkf{F7nYfDzo3Dc?C|>IVh9!pAUQegZOI7?DCrpA6vB-E%eO&ur(pOn{o!RC5xLH8Z$F)YFSCltVig7POh(f$X^ypw%hM4O6(SgMsFk^&fVn@>C>Xt-HZse6$cT z@&LP_s774t@x8;}biWk%iBHl4lqxlwDJNTx7`5w5-H?UB_D9Xs@M~cAyWkm$T|K6& zj|HvJ5OLI5Lfv=Bmk=egYHbOc*a*ZpUjt>W7>eicoHGN(um&(U;Qg4kJ_a;+jhOla zr!&WJkUpVq`o$kRox17#DqZku+$8)zi|txeepmS~`om#I0jBpWb3gz#+`y#L*x1-P zI?6Uw=6!R?otl=GqmvH151TmHI432n4~Io%Utfq_pUb5S5oy=kgsJ!fB|(^GSnZvd zBc(Wad!f3(6#L2#P3&P8rFZ~wJblw(rwa&w?frB7kg?UU#-LWpDDr-L9%ZeFB9mA|j#zAcbMt zt6qQhB7xS}I*@D=uzDFC0fh2s9x;I#BLcsBKpr*@#m*4{HDAc$jkB@79wj<6(@D&2 zB}NqxsyYUU3$PI+3B=7DPIpvcs%dt1cB|?`HL%A8QyyHzd_r=s4IM)l4TyQ$Z9&%^ zJTQi{6NFKbVB1W-%$Qk=i;L9y&tUR(-|-=rklK<}BV*W9)hUE%QVgu~eY9@-8U_&1 zpjpuX8gVYAhi3M$@4=&CeEZzt00945@UD@9wTU9T(8g*nSwJ&cD%9+$6@O$h%Vjf$l^lOg zvYA*6Az%_u= z7$W2AV1>25KWG3q&~g704~+pwa&%0L;Q+C@nHd!~H~t%GJG@N)Dx^l7(5eh9YiV#A z#b^JJ&m=fFI22up*t~>w#i3E^wA5;muGw%%)eqR=zP{_ygB1ogbW`y$ z#0^3y73InBw;%ow(b(Z?yF&UHm0{h2-Ace0(3&}d?D);!BM=fi12~I7tVq@T`+J35 z_B}S}|6}1_dH)^Zq3_h}^ZdmQy>XvZpM;$s2j)ErZD}bqd%F%`LBsy2u2(n@>dW%N! zB0ub#!lE;8Nz4c=$6hVeZjsL#8W_DfdcDx{md5Y$V$uW?kc+wf^du7k(w(>~i^D}f_)e^mm_83sA(f5u&V_V^+ zVl?ZfeNO+@x)Qb2T40pOl&87E6oC;%l5&6v9BK3e3W7ZT<=EpBodEyaGr)%K%qzl*go zL)y4c+D~mKnDK7(i2UhP(6fD;I12Z)Nm|z2pgzOuWj)lVzNZ&X35+Y#y!z5!UIN63 z++B6h*@K&an?jP6ihX|frOm=^b@c+*LzmgIh*{-wXf@5ZE%)W;EO%a)URQJilowXj z?fq5jhZd7nZ6f<>b27`+EWhpC_v(K?vv^EeSTS9wjxwxPK5%?hYduL2Rc9NbC+z*@ zxhsPD$r^g)G2*%;+D~yX)8osuOk<6B0Es`aeFNJKJBCgs`(x_$YIC@`e`edp5vf!~!4!3J_JT-mb~ z{c`wUZ=eMkl`g$b5#H|1RO#r_yXkzs3aBUPs94>j9oO&hKO|CEeShGKWjP+tdRV#X z4O|M=GZNk$&lD-{8I#UxXi8qyq@YS;Mq4a(tk_WRJeCIs&@zYq3C?uK<}Gj29>Xo4-Rcgwm@ zVLgM$*FoN4xEtQ7jTF>PSl6Cy~`=r70uiMOtS)JuyqV`(%m+-F>kQZ2vp9JHi{) z;a^ye%R2__)&hHAI=UbnxLT&VV2k31&5-e40shDj>z=2g{iwE^uw`7V{sX=Ac{?Fd zs#8OpBYmsdkeDVXlw*#(uhy01Qnh4%d~i@pt1= z`3lM5o;AM;37Vx6xjcU(@>pg=8@Hh>0?974!{=R)Kx8T|K8Xxml5ONqEAsW1Rz5$; zZDaMDKfmR3fV-NO=JAZgYYke8?r1VNuUq(jpC4OS z7^$8R(>zU|(JfjvG!&??UleFAnH^umZ!vb;vHPV|oS@iu`Fs9J$>fihSN4PscD(U} ztm4ZaxXSsy*-OqIh!e3}d%yBqa-dVikr#{k6t?1eF{zMlR8Q;H+t2U_) z{W_7L+0UUVskLb2W+RQHpfK>&q$wogPsy+KoJryNR{ovY4z^ zV>SC*jEq-TmGoQ}r7~Y{3|+R?@|7<%LPW`g!@b(uo8>`u)1}r7`NU0HtWyK`n~|0x zis~3IY?rp?cFj1)4EVeko#my(W?8q=m4&>>{*o+Op+}G52;fHxHHqMePX{aYU~b>$ zMW+^oq91O)XWg9kzvw`092JBD0y2pb|AsBF6a68XazanlVu4Nn=V97fK4mN74(0!T??oOCY*09L>4wRHMD$GWgTFSIq zWh-!f_C1de(Qhud@|H387DKwkm&W#|WCV_DE}W`vcm`CLb}-2lnl7d*l~|nkRz+=4v%UYgh#Rs(@a+`G zd80(-%UBGx!2p5Wliyp!3Q5!UHQ<@dUHJuu&m zE1H6#K3o0x&)#=X-|+ixJ@Aj~BSLPUATnG&r>_>eBZuPBCn??J3K_vUhy4!MJ*bPe z%uJy_T0L=L@~{16+Vy?84CJfbQ3VD=J+C}+rQ|TkE(y}3b|c%Ccl(e|-)!&uos*g0 zx*{W7w&m(0E-IEqwK@zM-K<$e9qEj)EFi2xkY0;{=@B*aI1y$^M>x?0Vsq8y7*zeQ zUUjVPOqW!Da4O$8DJHA#+b=EL_M*kslrsSmJ{qkymg$g{Skqy}0ykS3cbjmnDhaLM zJT~)%eRytxi2e&x#9bV26`B@>diW;$f37OI%|7qF!&K58#o3)M)YGl_=1t2_6iLDz zTuPZ5?eTh+jfLf*_KdB6hP|*abDXLPhs?wU4L55HBvWb?3oSkq4xtmLXR{18EPg(X zPLD}nu>6`XqTDc+%r=zLTy6Rgk4f$1%U}?qbF9u=I(<2*-&%FdvtK#G;#3{xnk8ix zjsDc3ox$*lPgke?ktCKHEv#A%I}gdui$~l+Q4pVifRXtwtM9IE9@^z;muu&2Fv; z@hhJwL3e`@L$=mMFDYlE`uer|c5M}*2*XdUFXrdwin!lpbd@hw!Q#XD25;FY$yC9& zWF{=jm3E^$wQqtWXj6p@seX;7fB0zCnq5=FX*KO5q2JIjdYeIb!HRROdfI-q=t6Z1 zUfF9|iYw79TD6Dw9fCvp3ZK=1;=tiaRZ(VJGUHJ-i|!?)FZS|L3K#nOnmF92kr~nI zJhJCke2s+$ER~+hTxcNBRw#fvky$UuZ+gqSBFGSy7leeQw~kl&dQYQyAS5uUPOt>U zZUw%LnAL4=hdg|AV!?TGb3A--SYE7+f7tUVY~-cVMQ*8;h{)Et2a1e9GHJEVhSQ?t zk784qTFbfD`f*kMpDJ{oSC9#x9)iGGvB}J}_3Tr?`Bvbo#Oohfa7cS>2I@0J#TVDN zsK?3gK$zQAnw$#BQ0joj!@wS?Qkz zG1&_Uw8xJ)96Kc7R!>$f{OI8cz9#rD{H*NzX2#|> z&zQeYR{X(tetxYwG8?daZ7+iUX+{+PdX8T(U>$O7X+HMT4Kw1`!%GVd7T6_&@!9!{ zJvxWFzdVK`-aZt(S-N5R);{u)ebC;HhlQd`2*i7^d+H2COHyH`Ah)*9m#x#hddhs) zg~ah@GCC&KjW^JXE7_f6itfpZv6q8HU~Vnb`UAn)YUHAaCC~5QzZ3y`(V<`SMnw|w zTkvNNDk#@So~)=LM`ww2*e7p0l^VGszca(>zO9e5yprGPMimbBl`4G<+xF;wQ0PBO zv)I;q&B%xBAAEssAo+JQBvb70fdhuYA>T3U-#A0DlN}xd?%^@L2KL|QMXx9X4t?lP zb_avy-=@J}XTl;$x-0zt)?kMN)?k|zLL;r{f6u{VkAOoE>lQ*&3;O$9(eR==-@;+C X Date: Thu, 25 Aug 2022 16:58:20 -0700 Subject: [PATCH 07/57] added error messages page and link fixes to a few pages --- website/content/docs/nia/usage/errors-ref.mdx | 81 +++++++++++++++++++ .../content/docs/nia/usage/requirements.mdx | 2 +- website/content/docs/nia/usage/run-ha.mdx | 2 +- website/data/docs-nav-data.json | 4 + 4 files changed, 87 insertions(+), 2 deletions(-) create mode 100644 website/content/docs/nia/usage/errors-ref.mdx diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx new file mode 100644 index 0000000000..c4aa3daa7b --- /dev/null +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -0,0 +1,81 @@ +--- +layout: docs +page_title: Error Messages +description: >- + Look up Consul-Terraform-Sync error message to learn how to resolve potential issues using CTS. +--- + +# Error Messages + +This topic explains error messages you may encounter when using Consul-Terraform-Sync (CTS). + +## Example error log messages + +If you configured the CTS cluster to run in [high availability mode](/docs/nia/usage/run-ha) and the the local module is missing, then the following message appears in the log: + +```shell-session +[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" +``` + +The resolution is to create the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. + +## Example API and CLI error messages + +In the following example response error: + +* CTS can determine the leader. +* `high_availability.instance.address` is configured for the leader. +* The CTS instance you sent the request to is not the leader. + + +```json +{ + "error": { + "message": "redirect requests to leader 'cts-01' at cts-01.example.com:8558" + } +} +``` + +The resolution is to redirect the request to the leader instance, for example: + +```shell-session +$ curl --request GET cts-01.example.com:8558/v1/tasks +``` +--- +In the following example response error: + +* CTS can determine the leader. +* The CTS instance you sent the request to is not the leader. +* `high_availability.instance.address` is not configured. + +```json + +{ + "error": { + "message": "redirect requests to leader 'cts-01'" + } +} +``` +The resolution is to identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. + +--- + +In the following example response error: + +* The CTS instance you sent the request to is not the leader. +* The CTS is unable to determine the leader. +* Note that these conditions are rare. + +```json +{ + "error": { + "message": "redirect requests to leader" + } +} +``` +The resolution is to identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: + +```shell-session +[INFO] ha: acquired leadership lock: id= +``` + diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx index 31e8ac186f..46c20da43c 100644 --- a/website/content/docs/nia/usage/requirements.mdx +++ b/website/content/docs/nia/usage/requirements.mdx @@ -89,7 +89,7 @@ The Terraform module for a task in CTS is the core component of the integration. Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. -Refer to [Configuration](/docs/nia/installation/configure) for information about configuring CTS and how to use Terraform providers and modules for tasks. +Refer to [Configuration](/docs/nia/configuration) for information about configuring CTS and how to use Terraform providers and modules for tasks. ### Partner Terraform Modules diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 50e8dda098..7f8da2a750 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -84,7 +84,7 @@ If the `high_availability.cluster.storage.namespace` field is configured, then y We recommend deploying a cluster that includes three CTS instances. This is so that the cluster has one leader and two followers. -1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/installation/configure) for all configuration options. +1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/configuration) for all configuration options. 1. Issue the startup command and pass the configuration file. Refer to the [`start` command reference](/docs/nia/cli/start#modes) for additional information about CTS startup modes. ```shell-session $ consul-terraform-sync start -config-file ha-config.hcl diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index eedbd4d9be..446a835640 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -774,6 +774,10 @@ { "title": "Run Consul-Terraform-Sync with High Availability", "path": "nia/usage/run-ha" + }, + { + "title": "Error Messages", + "path": "nia/usage/errors-ref" } ] }, From 0fe310605a3ea409828f1d6d65483212be298a6e Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 17:18:06 -0700 Subject: [PATCH 08/57] updates to the architecture page --- website/content/docs/nia/architecture.mdx | 67 +++++++++++++++-------- 1 file changed, 43 insertions(+), 24 deletions(-) diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index a6712a9733..b112b3a1b8 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -2,37 +2,29 @@ layout: docs page_title: Architecture description: >- - Consul-Terraform-Sync Architecture + Learn about the Consul-Terraform-Sync architecture and high-level CTS components, such as the Terraform driver and tasks. --- # Consul-Terraform-Sync Architecture -Consul-Terraform-Sync (CTS) is a service-oriented tool for managing -network infrastructure near real-time. CTS runs as a daemon -and integrates the network topology maintained by your Consul cluster with your -network infrastructure to dynamically secure and connect services. +Consul-Terraform-Sync (CTS) is a service-oriented tool for managing network infrastructure near real-time. CTS runs as a daemon and integrates the network topology maintained by your Consul cluster with your network infrastructure to dynamically secure and connect services. -## 10,000 Foot View +## CTS workflow + +The following diagram shows the CTS workflow as it monitors the Consul service catalog for updates. [![Consul-Terraform-Sync Architecture](/img/nia-highlevel-diagram.svg)](/img/nia-highlevel-diagram.svg) -The diagram shows CTS monitoring the Consul service catalog -for updates and utilizing Terraform to update the state of the infrastructure. +1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and Views](#watcher-and-views). +1. CTS detects a change. +1. CTS prompts Terraform to update the state of the infrastructure. -There are two principal aspects of Sync to know about corresponding to the -lines to Consul and Terraform in the diagram above. The line to Consul -represents the Watchers monitoring the state of Consul's service catalog (and -possibly KV store) while the line to Terraform represents tasks being run to -update the infrastructure. -## Watcher and Views +## Watcher and views -CTS monitors Consul for updates utilizing Consul's [Blocking -Queries](/api-docs/features/blocking) whenever supported, falling back on -polling when not. The watcher maintains a separate thread (known internally as -a view) for each value monitored, running any tasks that depend on that watched -value whenever it's updated. Say, for example, running a task to update a -proxy when an instance goes unhealthy. +CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as watchers. + +The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it's updated. These threads are referred to as views. For example, a thread may run a task to update a proxy when the watcher sees that an instance changes to an unhealthy state. ## Tasks @@ -53,8 +45,35 @@ network infrastructure. The following [drivers](/docs/nia/network-drivers#terraf Each driver includes a set of providers that [enables support](/docs/nia/terraform-modules) for a wide variety of infrastructure applications. -## Security Guidelines +## State storage and persistence -The [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) -tutorial contains a checklist of best practices to secure your -CTS installation for a production environment. +The following types of state information are associated with CTS. + +### Terraform state information + +By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). If the backend is not configured to a local location, then the data persists if CTS stops. + +### CTS task and event data + +By default, CTS stores task and event data in the Consul KV. This data is transient and does not persist unless you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha). High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. + +The data stored when operating in high availability mode includes task changes made using the task API or CLI, such as creating a new task, deleting a task, or enabling/disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). + +## Instance compatibility checks (high availability) + +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently, enabling CTS to properly perform automations configured in the state storage. + +The only incompatibility CTS checks for are tasks that are configured with a [local module](/docs/nia/configuration#module). CTS instances that do not include this module directory are incompatible. Example log: + +```shell-session +[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" +``` +Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information. + +CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. + +CTS will continue to run and log the error message if it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility will not run successfully. This is because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. + +## Security guidelines + +We recommend following the network security guidelines described in the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. The tutorial contains a checklist of best practices to secure your CTS installation for a production environment. From da0bbd80e8192795dadeffa69772466a00bc03cd Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 17:59:49 -0700 Subject: [PATCH 09/57] added ha information to the API endpoints docs --- website/content/docs/nia/api/status.mdx | 4 ++-- website/content/docs/nia/api/tasks.mdx | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index edfd512072..a129aa3215 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -6,9 +6,9 @@ description: >- --- # Status -The `/status` endpoints share status-related information for tasks. This information is available for understanding the status of individual tasks and across tasks. +The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). -The health status value is determined by aggregating the success or failure of the event of a task detecting changes in Consul services and then updating network infrastructure. Currently, only the 5 most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Overall Status diff --git a/website/content/docs/nia/api/tasks.mdx b/website/content/docs/nia/api/tasks.mdx index b870d26718..0e871224d8 100644 --- a/website/content/docs/nia/api/tasks.mdx +++ b/website/content/docs/nia/api/tasks.mdx @@ -9,6 +9,8 @@ description: >- The `/tasks` endpoints interact with the tasks that Consul-Terraform-Sync (CTS) is responsible for running. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. + ## Get Tasks This endpoint returns information about all existing tasks. From 7ab3a3052a57cadabd93f2fcf7734565794daa31 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 19:34:22 -0700 Subject: [PATCH 10/57] add HA to configuration reference, updated other configuration params --- website/content/docs/nia/configuration.mdx | 58 +++++++++++++++++----- 1 file changed, 45 insertions(+), 13 deletions(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 69d8111369..9be8a176e1 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -5,11 +5,11 @@ description: >- Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the consul-terraform-sync daemon. --- -# Configuration Options for Consul-Terraform-Sync +# Consul-Terraform-Sync Configuration -The Consul-Terraform-Sync (CTS) daemon is configured using configuration files and supports [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) (HCL) and JSON file formats. +This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL and JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. -## Global Config Options +## Global configurations Top level options are reserved for configuring CTS. @@ -44,7 +44,7 @@ tls { - `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - `log_level` - (string: "INFO") The log level to use for CTS logging. This defaults to "INFO". The available log levels are "TRACE", "DEBUG", "INFO", "WARN", and "ERR". - `port` - (int: 8558) The port for CTS to use to serve API requests. -- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. Used as the service ID for CTS if service registration is enabled. +- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in the cluster. - `syslog` - Specifies the syslog server for logging. - `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging. - `facility` - (string: "local0") Name of the syslog facility to log to. @@ -82,7 +82,7 @@ license { | `auto_retrieval` | Optional | object | Configures the license auto-retrieval used by CTS. To learn more, review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for details | Review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for defaults. | -#### Auto-Retrieval +### Auto-retrieval You can use the `auto_retrieval` block to configure the the automatic license retrieval in CTS. When enabled, CTS attempts to retrieve a new license from its configured Consul Enterprise backend once a day. If CTS cannot retrieve a license and the current license is reaching its expiration date, CTS attempts to retrieve a license with increased frequency, as defined by the [License Expiration Date Handling](/docs/nia/enterprise/license#license-expiration-handling). @@ -92,7 +92,7 @@ You can use the `auto_retrieval` block to configure the the automatic license re | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | string | If set to true, enables license auto-retrieval | true | -## Consul +## Consul connection The `consul` block configures the CTS connection with a Consul agent so that CTS can perform queries for task execution. It also configures the automatic registration of CTS as a service with Consul. @@ -127,7 +127,7 @@ consul { | `transport` | Optional | [transport](/docs/nia/configuration#transport) | Low-level network connection details || | `service_registration` | Optional| [service_registration](/docs/nia/configuration#service-registration) | Options for how CTS will register itself as a service with a health check to Consul. || -##### ACL Requirements +### ACL requirements The following table describes the ACL policies required by CTS. For more information, refer to the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync) tutorial. | Policy | Resources | @@ -140,7 +140,7 @@ The following table describes the ACL policies required by CTS. For more informa | `keys:write` | `consul-terraform-sync/` Only required when using Consul as the [Terraform backend](/docs/nia/configuration#backend). | -#### Auth +### Auth Configures HTTP basic authentication for communicating with Consul. | Parameter | Required | Type | Description | Default | @@ -149,7 +149,8 @@ Configures HTTP basic authentication for communicating with Consul. | `username` | Optional | string | Username for authentication| none | | `password` | Optional | string | Password for authentication| none | -#### TLS +### TLS + Configure TLS to use a secure client connection with Consul. Using HTTP/2 can solve issues related to hitting Consul's maximum connection limits, as well as improve efficiency when processing many blocking queries. This option is required for Consul-Terraform-Sync when connecting to a [Consul agent with TLS verification enabled for HTTPS connections](/docs/agent/config/config-files#verify_incoming). If Consul is using a self-signed certificate that you have not added to the global CA chain, you can set this certificate with `ca_cert` or `ca_path`. Alternatively, you can disable SSL verification by setting `verify` to false. However, disabling verification is a potential security vulnerability. @@ -164,7 +165,7 @@ If Consul is using a self-signed certificate that you have not added to the glob | `key` | Optional | string | The path to the PEM-encoded private key file used with the client certificate configured by `cert`. Required if Consul has `verify_incoming` set to true.

Can also be provided through the `CONSUL_CLIENT_KEY` environment variable. | none | | `server_name` | Optional | string | The server name to use as the Server Name Indication (SNI) for Consul when connecting via TLS.

Can also be provided through the `CONSUL_TLS_SERVER_NAME` environment variable. | none | -#### Transport +### Transport Configures the low-level network connection details to Consul. To achieve the shortest latency between a Consul service update to a task execution, configure `max_idle_conns_per_host` equal to or greater than the number of services in automation across all tasks. This value should be lower than the configured [`http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the Consul agent. @@ -182,15 +183,16 @@ If `max_idle_conns_per_host` and the number of services in automation is greater | `tls_handshake_timeout` | Optional | string | The amount of time to wait to complete the TLS handshake. | `10s`| -#### Service Registration +### Service registration CTS automatically registers itself with Consul as a service with a health check, using the [`id`](/docs/nia/configuration#id) configuration as the service ID. CTS deregisters itself with Consul when CTS stops gracefully. If CTS is unable to register with Consul, then it will log the error and continue without exiting. Service registration requires that the [Consul token](/docs/nia/configuration#consul) has an ACL policy of `service:write` for the CTS service. | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | -| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul.| `true` | -| `service_name` | Optional | string | The service name for CTS. | `consul-terraform-sync` | +| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. +| `true` | +| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high_availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | @@ -213,6 +215,35 @@ The default health check is an [HTTP check](/docs/discovery/checks#http-interval | Timeout | `2s` | | TLSSkipVerify | `false` | +## High availability + +Add a `high_availability` block to your configuration to enable CTS to run in high availability mode. Refer to [Run Consul-Terrform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. The `high_availability` block contains the following configuration items. + +### `high_availability.cluster` + +The `cluster` parameter contains configurations for the cluster you want to operate with high availability enabled. You can configure the following options: + +| Parameter | Description| Required | Type | +| --------- | ---------- | -------- | ------| +| `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information | Optional | Object | + +#### High availability cluster storage + +The `high_availability.cluster.storage` object contains the following configurations. + +| Parameter | Description| Required | Type | +| --------- | ---------- | -------- | ------| +| `parent_path` | Defines a parent path in the Consul KV for CTS to store state information. Default is `consul-terraform-sync/`. CTS automatically appends the cluster name to the parent path, so the effective default directory for state information is `consul-terraform-sync/`. | Optional | String | +| `namespace` | specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | +| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | `15s` | + +### `high_availbility.instance` + +The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: +* `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. + + ## Service ~> Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. See [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. @@ -624,6 +655,7 @@ driver "terraform" { - Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html). Visit the Terraform documentation links for details on backend configuration options. - If omitted, CTS will generate default values and use configurations from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html), which stores Terraform statefiles in the Consul KV. The [ACL token provided for Consul authentication](#consul) is used to read and write to the KV store and requires [Consul KV privileges](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifier appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. - The remote enhanced backend is not supported with the Terraform driver to run operations in Terraform Cloud. Use the [Terraform Cloud driver](#terraform-cloud-driver) to integrate CTS with Terraform Cloud for remote workspaces and remote operations. + - The `local` backend type is not compatible with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. Refer to [Instance compatibility checks](/docs/nia/architecture#instance-compatibility-checks-high-availability) for additional information. - `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the CTS log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. - `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the CTS daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path. - `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. From 8405908bcafb1111c6ed1b5f05749ac98e6291c6 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 19:36:54 -0700 Subject: [PATCH 11/57] updated the compatibility table for 0.7 --- website/content/docs/nia/compatibility.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/content/docs/nia/compatibility.mdx b/website/content/docs/nia/compatibility.mdx index c4cc632858..4443085af6 100644 --- a/website/content/docs/nia/compatibility.mdx +++ b/website/content/docs/nia/compatibility.mdx @@ -7,7 +7,7 @@ description: >- # Compatibility -The following tables list the Consul-Terraform-Sync (CTS) version compatibility for Consul, Terraform, and Vault. +This topic describes Consul-Terraform-Sync (CTS) cross-compatibility with Consul, Terraform, and Vault. ## Consul @@ -28,6 +28,7 @@ CTS integration with Terraform is supported for the following: | :------------------ | :-------------------- | :---------------------- | :--------------------------- | | CTS Enterprise 0.4+ | 0.13 - 1.1 | Latest | v202010-2 - Latest | | CTS Enterprise 0.3 | 0.13 - 1.1 | N/A | v202010-2 - Latest | +| CTS OSS 0.7 | 0.13 - 1.2 | N/A | N/A | | CTS OSS 0.3+ | 0.13 - 1.1 | N/A | N/A | | CTS OSS 0.2 | 0.13 - 1.0 | N/A | N/A | | CTS OSS 0.1 | 0.13 - 0.14 | N/A | N/A | From c56afd4913f7e2d21b2eba1edd0e22467f7346b5 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Fri, 26 Aug 2022 08:38:51 -0700 Subject: [PATCH 12/57] replaced HA diagrams with SVG files --- website/content/docs/nia/usage/run-ha.mdx | 4 ++-- website/public/img/nia/cts-ha-after.svg | 1 + website/public/img/nia/cts-ha-before.svg | 1 + 3 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 website/public/img/nia/cts-ha-after.svg create mode 100644 website/public/img/nia/cts-ha-before.svg diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 7f8da2a750..ef4b46eca0 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -21,11 +21,11 @@ In a standard configuration, CTS exits if errors occur when the CTS instance run The following diagram shows operating state when high availability is enabled: -![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.png) +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.svg) The following diagram shows the CTS cluster state after the leader stops. CTS Instance B becomes the leader responsible for monitoring and running tasks. -![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.png) +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.svg) ### Failover details diff --git a/website/public/img/nia/cts-ha-after.svg b/website/public/img/nia/cts-ha-after.svg new file mode 100644 index 0000000000..088089d205 --- /dev/null +++ b/website/public/img/nia/cts-ha-after.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/public/img/nia/cts-ha-before.svg b/website/public/img/nia/cts-ha-before.svg new file mode 100644 index 0000000000..546cf54893 --- /dev/null +++ b/website/public/img/nia/cts-ha-before.svg @@ -0,0 +1 @@ + \ No newline at end of file From 56470630fbb1d712746caed876c756b2ea2ed2be Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 11:58:19 -0700 Subject: [PATCH 13/57] Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/api/status.mdx | 2 +- website/content/docs/nia/api/tasks.mdx | 2 +- website/content/docs/nia/architecture.mdx | 14 ++++---- website/content/docs/nia/cli/start.mdx | 4 +-- website/content/docs/nia/configuration.mdx | 12 +++---- website/content/docs/nia/usage/errors-ref.mdx | 6 ++-- .../content/docs/nia/usage/requirements.mdx | 30 ++++++++-------- website/content/docs/nia/usage/run-ha.mdx | 34 +++++++------------ website/content/docs/nia/usage/run.mdx | 2 +- 9 files changed, 49 insertions(+), 57 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index a129aa3215..d364f93bec 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -8,7 +8,7 @@ description: >- The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Overall Status diff --git a/website/content/docs/nia/api/tasks.mdx b/website/content/docs/nia/api/tasks.mdx index 0e871224d8..60ca787a3b 100644 --- a/website/content/docs/nia/api/tasks.mdx +++ b/website/content/docs/nia/api/tasks.mdx @@ -9,7 +9,7 @@ description: >- The `/tasks` endpoints interact with the tasks that Consul-Terraform-Sync (CTS) is responsible for running. -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Get Tasks diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index b112b3a1b8..89514908e5 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -15,16 +15,16 @@ The following diagram shows the CTS workflow as it monitors the Consul service c [![Consul-Terraform-Sync Architecture](/img/nia-highlevel-diagram.svg)](/img/nia-highlevel-diagram.svg) -1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and Views](#watcher-and-views). +1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and views](#watcher-and-views). 1. CTS detects a change. 1. CTS prompts Terraform to update the state of the infrastructure. ## Watcher and views -CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as watchers. +CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as *watchers*. -The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it's updated. These threads are referred to as views. For example, a thread may run a task to update a proxy when the watcher sees that an instance changes to an unhealthy state. +The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it is updated. These threads are referred to as _views_. For example, a thread may run a task to update a proxy when the watcher detects that an instance has become unhealthy . ## Tasks @@ -51,17 +51,17 @@ The following types of state information are associated with CTS. ### Terraform state information -By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). If the backend is not configured to a local location, then the data persists if CTS stops. +By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). The data persists if CTS stops and the backend is configured to a remote location. ### CTS task and event data By default, CTS stores task and event data in the Consul KV. This data is transient and does not persist unless you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha). High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. -The data stored when operating in high availability mode includes task changes made using the task API or CLI, such as creating a new task, deleting a task, or enabling/disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). +The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). ## Instance compatibility checks (high availability) -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently, enabling CTS to properly perform automations configured in the state storage. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage. The only incompatibility CTS checks for are tasks that are configured with a [local module](/docs/nia/configuration#module). CTS instances that do not include this module directory are incompatible. Example log: @@ -72,7 +72,7 @@ Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. -CTS will continue to run and log the error message if it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility will not run successfully. This is because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. +CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This happens because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. ## Security guidelines diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 801b14ab6f..21730c3907 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -25,7 +25,7 @@ The following table describes all of the available flags. | --- | ---- | ---- | ---- | ---- | | `-config-dir`| Required when `-config-file` is not set | string | Specifies a directory containing CTS instance configuration files to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one directory of files. | none | | `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | -| `-inspect` | Optional | boolean | Starts CTS in inspect mode (refer to [Modes](#modes) for additional information). In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | +| `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | | `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | | `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | @@ -38,6 +38,6 @@ By default, CTS starts in long-running mode. The following table describes all a | Name | Description |How to start | | --- | --- | --- | | Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | -| Once mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | +| Once-mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | | Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| | High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 9be8a176e1..99c8b6f379 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -226,22 +226,22 @@ The `cluster` parameter contains configurations for the cluster you want to oper | Parameter | Description| Required | Type | | --------- | ---------- | -------- | ------| | `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | -| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information | Optional | Object | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | -#### High availability cluster storage +#### `high_availability.cluster.storage` The `high_availability.cluster.storage` object contains the following configurations. | Parameter | Description| Required | Type | | --------- | ---------- | -------- | ------| | `parent_path` | Defines a parent path in the Consul KV for CTS to store state information. Default is `consul-terraform-sync/`. CTS automatically appends the cluster name to the parent path, so the effective default directory for state information is `consul-terraform-sync/`. | Optional | String | -| `namespace` | specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | -| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | `15s` | +| `namespace` | Specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | +| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | String | ### `high_availbility.instance` The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: -* `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. +- `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. ## Service @@ -272,7 +272,7 @@ service { ## Task -A `task` block configures which task to execute in automation. When the task executes can be determined by the `condition` block. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). +A `task` block configures which task to execute in automation. Use the `condition` block to specify when the task executes. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). ```hcl task { diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index c4aa3daa7b..e7ca2e3631 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -23,9 +23,9 @@ The resolution is to create the missing local module on the incompatible CTS ins In the following example response error: -* CTS can determine the leader. -* `high_availability.instance.address` is configured for the leader. -* The CTS instance you sent the request to is not the leader. +- CTS can determine the leader. +- `high_availability.instance.address` is configured for the leader. +- The CTS instance you sent the request to is not the leader. ```json diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx index 46c20da43c..eeb6796649 100644 --- a/website/content/docs/nia/usage/requirements.mdx +++ b/website/content/docs/nia/usage/requirements.mdx @@ -9,13 +9,13 @@ description: >- The following components are required to run Consul-Terraform-Sync (CTS): -* A Terraform Provider -* A Terraform Module -* A Consul cluster running outside of the `consul-terraform-sync` daemon +- A Terraform provider +- A Terraform module +- A Consul cluster running outside of the `consul-terraform-sync` daemon You can add support for your network infrastructure through Terraform providers so that you can apply Terraform modules to implement network integrations. -The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). +The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driver has [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). ## Run a Consul cluster @@ -33,7 +33,7 @@ For information on compatible Consul versions, refer to the [Consul compatibilit The Consul agent must be running in order to dynamically update network devices. Refer to the [Consul agent documentation](/docs/agent/index) for information about configuring and starting a Consul agent. For hands-on instructions about running Consul agents, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). -When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitors a high number of services. +When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitoring a high number of services. To avoid reaching the limit prematurely, we recommend using HTTP/2 (requires HTTPS) to communicate between CTS and the Consul agent. When using HTTP/2, CTS establishes a single connection and reuses it for all communication. Refer to the [Consul Configuration section](/docs/nia/configuration#consul) for details. @@ -41,7 +41,7 @@ Alternatively, you can configure the [`limits.http_max_conns_per_client`](/docs/ ### Register services -CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. +CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon is operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. The following HTTP API request example registers a service named `web` with your Consul agent: @@ -68,7 +68,7 @@ For hands-on instructions on registering a service by loading a service definiti For production environments, we recommend operating a Consul cluster rather than a single agent. Refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started) for instructions on starting multiple Consul agents and joining them into a cluster. -## Network infrastructure (using a Terraform provider) +## Network infrastructure using a Terraform provider CTS integrations for the Terraform driver use Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver for CTS inherits the expansive collection of Terraform providers to integrate with. You can also specify a provider `source` in the [`required_providers` configuration](https://www.terraform.io/language/providers/requirements#requiring-providers) to use providers written by the community (requires Terraform 0.13 or later). @@ -78,16 +78,16 @@ To find providers for the infrastructure platforms you use, browse the providers ### How to create a provider -If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registery so that you can use it within a network integration task or createa compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: +If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registry so that you can use it within a network integration task or create a compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: -* [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) -* [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). +- [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) +- [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). -## Network integration (using a Terraform module) +## Network integration using a Terraform module -The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the condition under which your infrastructure is updated. +The Terraform module for a task in CTS is the core component of the integration. It declares which resources to use and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the conditions under which your infrastructure is updated. -Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. +Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver. You can also use a [module built by partners](#partner-terraform-modules). Refer to [Configuration](/docs/nia/configuration) for information about configuring CTS and how to use Terraform providers and modules for tasks. @@ -118,7 +118,7 @@ The modules listed below are available to use and are compatible with CTS. #### Citrix ADC -- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) +- Create, Update, and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) #### F5 @@ -126,7 +126,7 @@ The modules listed below are available to use and are compatible with CTS. #### NS1 -- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) +- Create, Delete, and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) #### Palo Alto Networks diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index ef4b46eca0..ba8494b85c 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -11,7 +11,7 @@ This topic describes how to run Consul-Terraform-Sync (CTS) configured for high ## Introduction -A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process if it is configured for high availability: +A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process when it is configured for high availability: 1. The CTS cluster promotes a new leader from the pool of followers in the network. 1. The new leader begins running all existing tasks in `once-mode` in order to process changes that occurred during the failover transition period. In this mode, CTS runs all existing tasks one time. @@ -29,11 +29,11 @@ The following diagram shows the CTS cluster state after the leader stops. CTS In ### Failover details -* The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. -* If failover occurs during task execution, a new leader is elected and will attempt to run all tasks once before continuing to monitor for changes. -* If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. -* If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. -* If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. +- The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. +- If failover occurs during task execution, a new leader is elected. The new leader will attempt to run all tasks once before continuing to monitor for changes. +- If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. +- If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. +- If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. Note that driver behavior is consistent whether or not CTS is running in high availability mode. @@ -46,7 +46,7 @@ Note that driver behavior is consistent whether or not CTS is running in high av You must configure appropriate ACL permissions for your cluster. Refer to [ACL permissions](#) for details. -It’s not required, but we recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. +We recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. ## Configuration @@ -97,7 +97,7 @@ We recommend deploying a cluster that includes three CTS instances. This is so t ## Modify an instance configuration -You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings, when high availability is enabled. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). +You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). 1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: ```shell-session @@ -111,7 +111,9 @@ You can implement a rolling update to update a non-task configuration for a CTS ## Modify tasks -When high availability is enabled, CTS persists task and event data (refer to State storage and persistence for additional information). You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes. This is to limit inconsistencies between the state and the configuration that can occur when mixing methods. +When high availability is enabled, CTS persists task and event data. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. + +You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes because inconsistencies between the state and the configuration can occur when mixing methods. ### Delete and recreate the task (recommended) @@ -137,19 +139,9 @@ Use the CTS API to identify the CTS leader instance and delete and replace a tas --data @payload.json \ localhost:8558/v1/tasks ``` + You can also use the [`task-create` command](/docs/nia/cli/task#task-create) to complete this step. -Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. - -```shell-session -$curl --header "Content-Type: application/json" \ ---request POST \ ---data @payload.json \ - localhost:8558/v1/tasks -``` - -You can also use the `task-create` command to complete this step. - ### Discard data with the `-reset-storage` flag You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/options) to discard persisted data if you need to update a task. @@ -181,4 +173,4 @@ Use the following troubleshooting procedure if a previous leader had been runnin 1. Check for differences between the previous leader and new leader, such as differences in configurations, environment variables, and local resources. 1. Start a new instance with the fix that resolves the issue. 1. Tear down the leader instance that has the issue and any other instances that may have the same issue. -1. Restart the instance(s) to implement the fix. +1. Restart the affected instances to implement the fix. diff --git a/website/content/docs/nia/usage/run.mdx b/website/content/docs/nia/usage/run.mdx index a90cfae1af..ef26f96c56 100644 --- a/website/content/docs/nia/usage/run.mdx +++ b/website/content/docs/nia/usage/run.mdx @@ -23,7 +23,7 @@ This topic describes the basic procedure for running Consul-Terraform-Sync (CTS) $ consul-terraform-sync start -config-file ``` -4. Check status of tasks. Replace port number if configured in Step 2. See additional API endpoints [here](/docs/nia/api) +4. Check status of tasks. Replace port number if configured in Step 2. Refer to [Consul-Terraform-Sync API](/docs/nia/api) for additional information. ```shell-session $ curl localhost:8558/status/tasks From 134f996df0e33c0a4f41dbd5edd40ddd352878ff Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 12:18:28 -0700 Subject: [PATCH 14/57] Apply suggestions from code review Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/configuration.mdx | 8 ++++---- website/content/docs/nia/usage/run-ha.mdx | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 99c8b6f379..6ad35577b6 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -7,7 +7,7 @@ description: >- # Consul-Terraform-Sync Configuration -This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL and JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. +This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL or JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. ## Global configurations @@ -219,7 +219,7 @@ The default health check is an [HTTP check](/docs/discovery/checks#http-interval Add a `high_availability` block to your configuration to enable CTS to run in high availability mode. Refer to [Run Consul-Terrform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. The `high_availability` block contains the following configuration items. -### `high_availability.cluster` +### High availability cluster The `cluster` parameter contains configurations for the cluster you want to operate with high availability enabled. You can configure the following options: @@ -238,7 +238,7 @@ The `high_availability.cluster.storage` object contains the following configurat | `namespace` | Specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | | `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | String | -### `high_availbility.instance` +### High availability instance The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: - `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. @@ -655,7 +655,7 @@ driver "terraform" { - Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html). Visit the Terraform documentation links for details on backend configuration options. - If omitted, CTS will generate default values and use configurations from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html), which stores Terraform statefiles in the Consul KV. The [ACL token provided for Consul authentication](#consul) is used to read and write to the KV store and requires [Consul KV privileges](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifier appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. - The remote enhanced backend is not supported with the Terraform driver to run operations in Terraform Cloud. Use the [Terraform Cloud driver](#terraform-cloud-driver) to integrate CTS with Terraform Cloud for remote workspaces and remote operations. - - The `local` backend type is not compatible with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. Refer to [Instance compatibility checks](/docs/nia/architecture#instance-compatibility-checks-high-availability) for additional information. + - The `local` backend type is not supported with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. - `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the CTS log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. - `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the CTS daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path. - `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index ba8494b85c..87c6c68447 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -50,7 +50,7 @@ We recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-clo ## Configuration -Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](#) for details about the configuration fields for the `high_availability` block. +Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](/docs/nia/configuration#high-availability) for details about the configuration fields for the `high_availability` block. The following example configures high availability functionality for a cluster named `cts-cluster`: @@ -117,9 +117,9 @@ You can use the following methods for modifying tasks when high availability is ### Delete and recreate the task (recommended) -Use the CTS API to identify the CTS leader instance and delete and replace a task. +Use the CTS API to identify the CTS leader instance as well as delete and replace a task. -1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: +1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= @@ -144,7 +144,7 @@ Use the CTS API to identify the CTS leader instance and delete and replace a tas ### Discard data with the `-reset-storage` flag -You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/options) to discard persisted data if you need to update a task. +You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/start#options) to discard persisted data if you need to update a task. 1. Stop a follower instance. 1. Update the instance’s task configuration. From 8593e712c16d8e9ff5e6e2b09d1275d87f7f1514 Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 12:20:29 -0700 Subject: [PATCH 15/57] Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> --- website/content/docs/nia/cli/start.mdx | 2 +- website/content/docs/nia/compatibility.mdx | 1 + website/content/docs/nia/usage/errors-ref.mdx | 1 - 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 21730c3907..9b57be03fc 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -27,7 +27,7 @@ The following table describes all of the available flags. | `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | | `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | -| `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | +| `-once` | Optional | boolean | Starts CTS in once-mode. In once-mode, CTS renders templates, runs tasks once, and disables buffer periods. Refer to [Modes](#modes) for additional information. | `false` | | `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | | `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | diff --git a/website/content/docs/nia/compatibility.mdx b/website/content/docs/nia/compatibility.mdx index 4443085af6..05e301e028 100644 --- a/website/content/docs/nia/compatibility.mdx +++ b/website/content/docs/nia/compatibility.mdx @@ -26,6 +26,7 @@ CTS integration with Terraform is supported for the following: | CTS Version | Terraform CLI Version | Terraform Cloud Version | Terraform Enterprise Version | | :------------------ | :-------------------- | :---------------------- | :--------------------------- | +| CTS Enterprise 0.7 | 0.13-1.2 | Latest | v202010-2 - Latest | | CTS Enterprise 0.4+ | 0.13 - 1.1 | Latest | v202010-2 - Latest | | CTS Enterprise 0.3 | 0.13 - 1.1 | N/A | v202010-2 - Latest | | CTS OSS 0.7 | 0.13 - 1.2 | N/A | N/A | diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index e7ca2e3631..a456736c2c 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -49,7 +49,6 @@ In the following example response error: * `high_availability.instance.address` is not configured. ```json - { "error": { "message": "redirect requests to leader 'cts-01'" From d5145629c3ddf8faf7b3951af23c8dd29956634c Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 14:28:20 -0700 Subject: [PATCH 16/57] Apply suggestions from code review Co-authored-by: lornasong --- website/content/docs/nia/architecture.mdx | 6 +++--- website/content/docs/nia/cli/start.mdx | 8 ++++---- website/content/docs/nia/configuration.mdx | 7 +++---- website/content/docs/nia/usage/errors-ref.mdx | 2 +- website/content/docs/nia/usage/run-ha.mdx | 8 +++++--- 5 files changed, 16 insertions(+), 15 deletions(-) diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index 89514908e5..39ba71f654 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -55,7 +55,7 @@ By default, CTS stores [Terraform state data](https://www.terraform.io/docs/stat ### CTS task and event data -By default, CTS stores task and event data in the Consul KV. This data is transient and does not persist unless you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha). High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. +By default, CTS stores task and event data in memory. This data is transient and does not persist. If you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha), CTS stores the data in the Consul KV. High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). @@ -63,7 +63,7 @@ The data stored when operating in high availability mode includes task changes m If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage. -The only incompatibility CTS checks for are tasks that are configured with a [local module](/docs/nia/configuration#module). CTS instances that do not include this module directory are incompatible. Example log: +The CTS instance compatibility check reports an error if the task [module](/docs/nia/configuration#module) is configured with a local module, but the module does not exist on the CTS instance. Refer to the [Terraform documentation](https://www.terraform.io/language/modules/sources#module-sources) for additional information about module sources. Example log: ```shell-session [ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" @@ -72,7 +72,7 @@ Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. -CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This happens because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. +CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This can happen when all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. ## Security guidelines diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 9b57be03fc..c47a183df6 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -28,7 +28,7 @@ The following table describes all of the available flags. | `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | | `-once` | Optional | boolean | Starts CTS in once-mode. In once-mode, CTS renders templates, runs tasks once, and disables buffer periods. Refer to [Modes](#modes) for additional information. | `false` | -| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | `false` | | `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | ## Modes @@ -37,7 +37,7 @@ By default, CTS starts in long-running mode. The following table describes all a | Name | Description |How to start | | --- | --- | --- | -| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | +| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process.

When the long-running process begins, the CTS daemon serves API and command requests.

If an error occurs, CTS logs it and continues running.

| No additional flags.
This is the default mode. | | Once-mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | -| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| -| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| +| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks will update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| +| High availability mode | A long-running process that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 6ad35577b6..28b6063b48 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -44,7 +44,7 @@ tls { - `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - `log_level` - (string: "INFO") The log level to use for CTS logging. This defaults to "INFO". The available log levels are "TRACE", "DEBUG", "INFO", "WARN", and "ERR". - `port` - (int: 8558) The port for CTS to use to serve API requests. -- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in the cluster. +- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in a high availability cluster. - `syslog` - Specifies the syslog server for logging. - `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging. - `facility` - (string: "local0") Name of the syslog facility to log to. @@ -190,8 +190,7 @@ Service registration requires that the [Consul token](/docs/nia/configuration#co | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | -| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. -| `true` | +| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. | `true` | | `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high_availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | @@ -226,7 +225,7 @@ The `cluster` parameter contains configurations for the cluster you want to oper | Parameter | Description| Required | Type | | --------- | ---------- | -------- | ------| | `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | -| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `"consul"` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | #### `high_availability.cluster.storage` diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index a456736c2c..2dc82d7925 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -17,7 +17,7 @@ If you configured the CTS cluster to run in [high availability mode](/docs/nia/u [ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" ``` -The resolution is to create the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. +The resolution is to add the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. ## Example API and CLI error messages diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 87c6c68447..52c2c77268 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -6,7 +6,9 @@ description: >- --- # Run Consul-Terraform-Sync with High Availability - + + Licenses are only required for Consul-Terraform-Sync (CTS) Enterprise + This topic describes how to run Consul-Terraform-Sync (CTS) configured for high availability. High availability is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. ## Introduction @@ -19,7 +21,7 @@ A network always has exactly one instance of the CTS cluster that is the designa In a standard configuration, CTS exits if errors occur when the CTS instance runs tasks in `once-mode`. In a high availability configuration, CTS logs the errors and continues to operate without interruption. -The following diagram shows operating state when high availability is enabled: +The following diagram shows operating state when high availability is enabled. CTS Instance A is the current leader and is responsible for monitoring and running tasks: ![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.svg) @@ -89,7 +91,7 @@ We recommend deploying a cluster that includes three CTS instances. This is so t ```shell-session $ consul-terraform-sync start -config-file ha-config.hcl ``` -1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. +1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Only the leader of the cluster will return a successful response. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. ```shell-session $ curl localhost:/status/tasks From dda670044e0bac044fda648e99e6c0bab8abd4c2 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Fri, 26 Aug 2022 14:46:13 -0700 Subject: [PATCH 17/57] final content tweaks for CTS 0.7 docs --- website/content/docs/nia/usage/errors-ref.mdx | 55 ++++++++++++------- website/content/docs/nia/usage/run-ha.mdx | 5 +- 2 files changed, 40 insertions(+), 20 deletions(-) diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 2dc82d7925..9c7cd8bf89 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -21,12 +21,7 @@ The resolution is to add the missing local module on the incompatible CTS instan ## Example API and CLI error messages -In the following example response error: - -- CTS can determine the leader. -- `high_availability.instance.address` is configured for the leader. -- The CTS instance you sent the request to is not the leader. - +**Error**: ```json { @@ -36,17 +31,25 @@ In the following example response error: } ``` -The resolution is to redirect the request to the leader instance, for example: +**Conditions**: + +- CTS can determine the leader. +- `high_availability.instance.address` is configured for the leader. +- The CTS instance you sent the request to is not the leader. + + + +**Resolution**: + +Redirect the request to the leader instance, for example: ```shell-session $ curl --request GET cts-01.example.com:8558/v1/tasks ``` --- -In the following example response error: -* CTS can determine the leader. -* The CTS instance you sent the request to is not the leader. -* `high_availability.instance.address` is not configured. + +**Error**: ```json { @@ -55,15 +58,20 @@ In the following example response error: } } ``` -The resolution is to identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. + +**Conditions**: + +* CTS can determine the leader. +* The CTS instance you sent the request to is not the leader. +* `high_availability.instance.address` is not configured. + +**Resolution**: + +Identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. --- -In the following example response error: - -* The CTS instance you sent the request to is not the leader. -* The CTS is unable to determine the leader. -* Note that these conditions are rare. +**Error**: ```json { @@ -72,9 +80,18 @@ In the following example response error: } } ``` -The resolution is to identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: + +**Conditions**: + +* The CTS instance you sent the request to is not the leader. +* The CTS is unable to determine the leader. +* Note that these conditions are rare. + + +**Resolution**: + +Identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= ``` - diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 52c2c77268..23ba1d414f 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -41,7 +41,8 @@ Note that driver behavior is consistent whether or not CTS is running in high av ## Requirements -* Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. +Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. + * CTS Enterprise 0.7 or later * Terraform CLI 0.13 or later * All instances in a cluster must be in the same datacenter. @@ -97,6 +98,8 @@ We recommend deploying a cluster that includes three CTS instances. This is so t $ curl localhost:/status/tasks ``` +Repeat the procedure to start the remaining instances for your cluster. + ## Modify an instance configuration You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). From 1913b16283b4af87bb376c9e6f3cc755f4ec9bd5 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Fri, 26 Aug 2022 14:53:12 -0700 Subject: [PATCH 18/57] redirects for CTS 0.7 --- website/redirects.js | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/website/redirects.js b/website/redirects.js index 76d18f6523..d6564754b8 100644 --- a/website/redirects.js +++ b/website/redirects.js @@ -1273,4 +1273,14 @@ module.exports = [ destination: '/docs/k8s/installation/vault/data-integration/connect-ca', permanent: true, }, + { + source: '/docs/nia/installation/run', + destination: '/docs/nia/usage/run', + permanent: true, + }, + { + source: '/docs/nia/installation/requirements', + destination: '/docs/nia/usage/requirements', + permanent: true, + }, ] From 211b593a5468b937d88d0d27ee0eb48adaf198a2 Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 15:48:55 -0700 Subject: [PATCH 19/57] Update website/content/docs/nia/usage/run-ha.mdx --- website/content/docs/nia/usage/run-ha.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 23ba1d414f..e7dac83c0a 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -98,7 +98,7 @@ We recommend deploying a cluster that includes three CTS instances. This is so t $ curl localhost:/status/tasks ``` -Repeat the procedure to start the remaining instances for your cluster. +Repeat the procedure to start the remaining instances for your cluster. We recommend using near-identical configurations for all instances in your cluster. You may not be able to use exact configurations in all cases, but starting instances with the same configuration improves consistency and reduces confusion if you need to troubleshoot errors. ## Modify an instance configuration From 8e76d38db9d4a44f4c021e3947734cca83ca0377 Mon Sep 17 00:00:00 2001 From: Tu Nguyen Date: Mon, 29 Aug 2022 15:33:22 -0700 Subject: [PATCH 20/57] Apply suggestions from code review --- website/content/docs/nia/api/status.mdx | 2 +- website/content/docs/nia/configuration.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index d364f93bec..c66a63735a 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -6,7 +6,7 @@ description: >- --- # Status -The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). +The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 90e5905153..3682d91fdd 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -245,7 +245,7 @@ The `instance` parameter is an object that contains configurations unique to the ## Service -~> Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. See [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. +~> **Note:** Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. Refer to [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. A `service` block is an optional block to explicitly define the services configured in the `task` block's `services` field (deprecated). `service` blocks do not define services configured in the `task` block's `condition "services"` or `module_input "services` blocks. From c76116fec2edd4ca51476f826f560480ee106f85 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 11:55:07 -0700 Subject: [PATCH 21/57] added cluster endpoint to status API docs --- website/content/docs/nia/api/status.mdx | 92 ++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 11 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index c66a63735a..12ca873467 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -10,24 +10,24 @@ The `/status` endpoints return status-related information for tasks. This endpoi If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. -## Overall Status +## Status for all tasks -This endpoint currently returns the overall status information for all tasks. +This endpoint returns the overall status information for all tasks. | Method | Path | Produces | | ------ | ------------------- | ------------------ | | `GET` | `/status` | `application/json` | -### Request Parameters +### Request parameters Currently no request parameters are offered for the overall status API. -### Response Statuses +### Response statuses | Status | Reason | | ------ | ---------------- | | 200 | Successfully retrieved the status | -### Response Fields +### Response fields | Name | Type | Description | | ----- | ------ | ------------------ | @@ -65,7 +65,7 @@ Response: } ``` -## Task Status +## Task status This endpoint returns the individual task status information for a single specified task or for all tasks. @@ -79,7 +79,7 @@ Task health status value is determined by the success or failure of all stored [ | ------ | ------------------- | ------------------ | | `GET` | `/status/tasks/:task_name` | `application/json` | -### Request Parameters +### Request parameters | Name | Required | Type | Description | Default | | -------- | -------- | ------ | ------------------ | ------- | @@ -87,14 +87,14 @@ Task health status value is determined by the success or failure of all stored [ |`include` | Optional | string | Only accepts the value `"events"`. Use to include stored event information in response. | none |`status` | Optional | string | Only accepts health status values `"successful"`, `"errored"`, `"critical"`, or `"unknown"`. Use to filter response by tasks that have the specified health status value. Recommend setting this parameter when requesting all tasks i.e. no `task` parameter is set. | none -### Response Statuses +### Response statuses | Status | Reason | | ------ | ---------------- | | 200 | Successfully retrieved the task status | | 404 | Task with the given name not found | -### Response Fields +### Response fields The response is a JSON map of task name to a status information structure with the following fields. @@ -126,7 +126,11 @@ Event represents the process of updating network infrastructure of a task. The d |`config.source` | string | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** Module configured for the task. |`config.providers` | list[string] | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** List of the providers configured for the task. -### Example: All Task Statuses +### Examples + +The following examples call `status` endpoints to retrieve information about tasks. + +#### All task statuses Request: ```shell-session @@ -163,7 +167,7 @@ Response: } ``` -### Example: Individual Task Status with Events +#### Individual task status with events Request: ```shell-session @@ -225,3 +229,69 @@ Response: } } ``` + +## Cluster status + +The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. + +| Method | Path | Response format | +| ------ | ----------------- | ------------------ | +| `GET` | `/status/cluster` | `application/json` | + +### Request parameters + +Currently no request parameters are offered for the overall status API. + +### Request statuses + +- `200`: Successfully retrieved the cluster status + +### Response fields + +| Field | Type | Description | +| --- | ---- | --- | +| `cluster_name` | string | Identifies the name of the cluster. | +| `members` | list | Contains the CTS instance objects. | +| `members.address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | +| `members.healthy` | Boolean | Indicates the health of the instance. | +| `members.id` | string | Indicates the instance ID. | +| `members.leader` | Boolean | Identifies the cluster leader. | +| `members.service_name` | string | Identifies the name of the service that the instance represents. The value should always be `consul-terraform-sync`. | +| `request_id` | string | Unique identifier for the request. | + +### Example + +The following command calls the `/status/cluster` endpoint: + +```shell-session +$ curl –request GET http://localhost:8553/v1/status/cluster +``` + +The following example response shows that there are three CTS instances. The cluster is located at `cts-02.example.com`. The leader instance, `cts-02`, is the only healthy instance: + +```json +{ + "cluster_name": "cluster-a", + "members": [ + { + "address": "cts-02.example.com", + "healthy": true, + "id": "cts-02", + "leader": true, + "service_name": "consul-terraform-sync" + }, + { + "healthy": false, + "id": "cts-03", + "leader": false, + "service_name": "consul-terraform-sync" + }, + { + "healthy": false, + "id": "cts-01", + "leader": false, + "service_name": "consul-terraform-sync" + } + ], + "request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88" +} From 47f3a00969c4912bb528ecd33e998eb113e54cbc Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 11:59:21 -0700 Subject: [PATCH 22/57] added no-HA API status error message to EMs page --- website/content/docs/nia/api/status.mdx | 2 +- website/content/docs/nia/usage/errors-ref.mdx | 20 +++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index 12ca873467..0f5d737619 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -232,7 +232,7 @@ Response: ## Cluster status -The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. +The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. | Method | Path | Response format | | ------ | ----------------- | ------------------ | diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 9c7cd8bf89..4f433c9ef8 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -95,3 +95,23 @@ Identify and send the request to the leader CTS instance. You can identify the l ```shell-session [INFO] ha: acquired leadership lock: id= ``` + +--- + +**Error**: + +```json +{ + "error": { + "message": "this endpoint is only available with high availability configured" + } +} +``` + +**Conditions**: + +- CTS is not configured for [high availability](/docs/nia/usage/run-ha). + +**Resolution**: + +Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). \ No newline at end of file From 321044312f2427a7c05a72350693a5dba9173a90 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 12:09:03 -0700 Subject: [PATCH 23/57] udpated how to ID the leader in the usage docs --- website/content/docs/nia/usage/errors-ref.mdx | 11 ++++++++--- website/content/docs/nia/usage/run-ha.mdx | 6 +++--- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 4f433c9ef8..01aa10b6fd 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -67,7 +67,12 @@ $ curl --request GET cts-01.example.com:8558/v1/tasks **Resolution**: -Identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. +Identify the leader instance address and redirect the request to the leader. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: + +```shell-session +[INFO] ha: acquired leadership lock: id=. + +We recommend deploying a cluster that has three instances. --- @@ -90,7 +95,7 @@ Identify the leader instance address and redirect the request to the leader. We **Resolution**: -Identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: +Identify and send the request to the leader CTS instance. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= @@ -110,7 +115,7 @@ Identify and send the request to the leader CTS instance. You can identify the l **Conditions**: -- CTS is not configured for [high availability](/docs/nia/usage/run-ha). +- You called the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) without configuring CTS for [high availability](/docs/nia/usage/run-ha). **Resolution**: diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index e7dac83c0a..81fa5c2d16 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -104,7 +104,7 @@ Repeat the procedure to start the remaining instances for your cluster. We recom You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). -1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: +1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= ``` @@ -121,10 +121,10 @@ When high availability is enabled, CTS persists task and event data. Refer to [S You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes because inconsistencies between the state and the configuration can occur when mixing methods. ### Delete and recreate the task (recommended) - + Use the CTS API to identify the CTS leader instance as well as delete and replace a task. -1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: +1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= From 120e8b43ec518872538fc7b41f0470c18c5fe6b5 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 16:00:09 -0700 Subject: [PATCH 24/57] applied AJs feedback and made a few additional improvements --- website/content/docs/nia/api/index.mdx | 27 +++++------ website/content/docs/nia/api/status.mdx | 46 +++++++++++-------- website/content/docs/nia/usage/errors-ref.mdx | 23 +++++++++- 3 files changed, 60 insertions(+), 36 deletions(-) diff --git a/website/content/docs/nia/api/index.mdx b/website/content/docs/nia/api/index.mdx index 3e38803027..07b1d396e6 100644 --- a/website/content/docs/nia/api/index.mdx +++ b/website/content/docs/nia/api/index.mdx @@ -2,32 +2,29 @@ layout: docs page_title: Consul-Terraform-Sync API description: >- - How to use the Consul-Terraform-Sync API + Consul-Terraform-Sync provides an API interface that lets you query CTS instance health, check CTS instance status, and run CTS tasks. --- -# Consul-Terraform-Sync API +# Consul-Terraform-Sync API Overview -When running in [daemon mode](/docs/nia/cli#daemon-mode), Consul-Terraform-Sync (CTS) serves an HTTP API interface. Details of the available API endpoints is available in the navigation to the left. +Consul-Terraform-Sync (CTS) provides an HTTP API interface for querying CTS instances and running and managing tasks. -### Port +## Port The API is served at the default port `8558` or a different port if set with [`port` configuration](/docs/nia/configuration#port) -### Version Prefix +## Version prefix All API routes are prefixed with `/v1/`. This documentation is for v1 of the API, which is the only version currently. Example: `localhost:8558/v1/status` -### Error +## Request ID -Successful API requests will receive a 2XX success status code. For other unsuccessful status codes, when possible, more details will be provided in a response body containing an error object. +Each call to a CTS API endpoint returns a `request_id` field. The field is a string generated by the API. Example: -Example: Status 400 Bad Request -```json -{ - "error": { - "message": "example error message: unsupported status parameter value" - } -} -``` +`"request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88"` + +## Error messages + +The API sends a response code in the 200 range if the call is successful. If the call is unsuccessful, the API sends an error message that includes additional information when possible. Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information. \ No newline at end of file diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index 0f5d737619..26459692d3 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -8,7 +8,7 @@ description: >- The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send a request to the [`/status/cluster` endpoint](#cluster-status) of any instance to get information about the cluster. Requests to the `status` endpoint return a 400 error, however, if high availability is enabled and the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Status for all tasks @@ -19,13 +19,12 @@ This endpoint returns the overall status information for all tasks. | `GET` | `/status` | `application/json` | ### Request parameters -Currently no request parameters are offered for the overall status API. + +No request parameters are available for this endpoint. ### Response statuses -| Status | Reason | -| ------ | ---------------- | -| 200 | Successfully retrieved the status | +- `200`: Successfully retrieved the status. ### Response fields @@ -89,10 +88,8 @@ Task health status value is determined by the success or failure of all stored [ ### Response statuses -| Status | Reason | -| ------ | ---------------- | -| 200 | Successfully retrieved the task status | -| 404 | Task with the given name not found | +- `200`: Successfully retrieved the task status +- `404`: Task with the given name not found ### Response fields @@ -232,7 +229,7 @@ Response: ## Cluster status -The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. +The `/v1/status/cluster` API endpoint returns information about high-availability clusters and its members, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. | Method | Path | Response format | | ------ | ----------------- | ------------------ | @@ -248,16 +245,24 @@ Currently no request parameters are offered for the overall status API. ### Response fields +The following table describes the responses that the `status/cluster` endpoint can send. + | Field | Type | Description | | --- | ---- | --- | | `cluster_name` | string | Identifies the name of the cluster. | -| `members` | list | Contains the CTS instance objects. | -| `members.address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | -| `members.healthy` | Boolean | Indicates the health of the instance. | -| `members.id` | string | Indicates the instance ID. | -| `members.leader` | Boolean | Identifies the cluster leader. | -| `members.service_name` | string | Identifies the name of the service that the instance represents. The value should always be `consul-terraform-sync`. | -| `request_id` | string | Unique identifier for the request. | +| `members` | array | Contains an array of [member objects](#member-objects). Each object in the `members` array represents a CTS instance. | + +#### Member objects + +The following table describes the fields available for objects in the `members` array. + +| Field | Type | Description | +| --- | ---- | --- | +| `address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured on the leader instance. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | +| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | +| `id` | string | Indicates the service registration ID. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | +| `leader` | boolean | Identifies the cluster leader. | +| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | ### Example @@ -267,7 +272,7 @@ The following command calls the `/status/cluster` endpoint: $ curl –request GET http://localhost:8553/v1/status/cluster ``` -The following example response shows that there are three CTS instances. The cluster is located at `cts-02.example.com`. The leader instance, `cts-02`, is the only healthy instance: +The following example response shows three CTS instances. The cluster leader is `cts-02` and is advertising that its address is `cts-02.example.com`: ```json { @@ -281,13 +286,13 @@ The following example response shows that there are three CTS instances. The clu "service_name": "consul-terraform-sync" }, { - "healthy": false, + "healthy": true, "id": "cts-03", "leader": false, "service_name": "consul-terraform-sync" }, { - "healthy": false, + "healthy": true, "id": "cts-01", "leader": false, "service_name": "consul-terraform-sync" @@ -295,3 +300,4 @@ The following example response shows that there are three CTS instances. The clu ], "request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88" } +``` \ No newline at end of file diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 01aa10b6fd..0b24a1189a 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -119,4 +119,25 @@ Identify and send the request to the leader CTS instance. You can identify the l **Resolution**: -Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). \ No newline at end of file +Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). + +--- + +**Error**: + +```json +{ + "error": { + "message": "example error message: unsupported status parameter value" + } +} +``` + +**Conditions**: + +- You sent a request to the `status` API endpoint. +- The request included an unsupported parameter value. + +**Resolution**: + +Send a new request and verify that all of the parameter values are correct. \ No newline at end of file From 87e4a66c0dbb4b61a28f54570332723eade4e830 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 16:13:28 -0700 Subject: [PATCH 25/57] fixed bad links in status/cluster endpoint ref --- website/content/docs/nia/api/status.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index 26459692d3..ad551e9ff9 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -259,10 +259,10 @@ The following table describes the fields available for objects in the `members` | Field | Type | Description | | --- | ---- | --- | | `address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured on the leader instance. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | -| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | -| `id` | string | Indicates the service registration ID. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | +| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | +| `id` | string | Indicates the service registration ID. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | | `leader` | boolean | Identifies the cluster leader. | -| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | +| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | ### Example From d974e1903cf10bb6cd1fe0ae7fc53450aadddffe Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Wed, 7 Sep 2022 12:53:58 -0700 Subject: [PATCH 26/57] Apply suggestions from code review Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> --- website/content/docs/nia/api/status.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index ad551e9ff9..108f2c9aa0 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -8,7 +8,7 @@ description: >- The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). -If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send a request to the [`/status/cluster` endpoint](#cluster-status) of any instance to get information about the cluster. Requests to the `status` endpoint return a 400 error, however, if high availability is enabled and the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send requests to the [`/status/cluster` endpoint path](#cluster-status) on any CTS cluster member instance to receive information about the entire cluster. Calling the `status` endpoint path (without `/cluster`), however, returns a 400 error if the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Status for all tasks @@ -237,7 +237,7 @@ The `/v1/status/cluster` API endpoint returns information about high-availabilit ### Request parameters -Currently no request parameters are offered for the overall status API. +Currently no request parameters are offered for the cluster status API. ### Request statuses @@ -250,7 +250,7 @@ The following table describes the responses that the `status/cluster` endpoint c | Field | Type | Description | | --- | ---- | --- | | `cluster_name` | string | Identifies the name of the cluster. | -| `members` | array | Contains an array of [member objects](#member-objects). Each object in the `members` array represents a CTS instance. | +| `members` | list[[member](#member-objects)] | Contains a list of [member objects](#member-objects). Each object in the `members` list represents a CTS instance. | #### Member objects @@ -262,7 +262,7 @@ The following table describes the fields available for objects in the `members` | `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | | `id` | string | Indicates the service registration ID. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | | `leader` | boolean | Identifies the cluster leader. | -| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | +| `service_name` | string | Identifies the name of the service that the instance represents. The value is set by the `service_name` field in the [Service Registration](/docs/nia/configuration#service-registration) configuration. | ### Example From 7ced09c8aadd44add27bccd064b3081112fae932 Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Wed, 7 Sep 2022 14:00:56 -0700 Subject: [PATCH 27/57] Apply suggestions from code review Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 3682d91fdd..d32a1fc97c 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -191,7 +191,7 @@ Service registration requires that the [Consul token](/docs/nia/configuration#co | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. | `true` | -| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high_availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | +| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high-availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | From fc04fb1b5193c0b647134042b75b06c3e3b236cc Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Wed, 7 Sep 2022 14:13:39 -0700 Subject: [PATCH 28/57] updated HA configuration ref --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index d32a1fc97c..36bff7f2dc 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -227,7 +227,7 @@ The `cluster` parameter contains configurations for the cluster you want to oper | `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | | `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `"consul"` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | -#### `high_availability.cluster.storage` +#### High availability cluster storage The `high_availability.cluster.storage` object contains the following configurations. From e8a03490b819ed734bc23ddd6294ca20e8e8f344 Mon Sep 17 00:00:00 2001 From: Noe Brown Date: Wed, 8 Jun 2022 12:11:15 -0400 Subject: [PATCH 29/57] Updated the default service name to lower case Changes to doc : Changed default service_name to consul-terraform-sync. Service name is used in service registration it was previously proper case, it is now lower case. --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index c8ae16e123..ed3fe2432a 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -190,7 +190,7 @@ Service registration requires that the [Consul token](/docs/nia/configuration#co | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul.| `true` | -| `service_name` | Optional | string | The service name for CTS. | `Consul-Terraform-Sync` | +| `service_name` | Optional | string | The service name for CTS. | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | From eab4a08417b97ab1caf52740ea61bf631fb1fce9 Mon Sep 17 00:00:00 2001 From: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Date: Fri, 15 Jul 2022 12:35:33 -0700 Subject: [PATCH 30/57] Nia/cts tasks not required (#13731) * update docs * Update website/content/docs/nia/configuration.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index ed3fe2432a..d57dde75b8 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -241,7 +241,7 @@ service { ## Task -A `task` block configures which task to execute in automation. When the task should execute can be determined by the `service` block (deprecated) and `condition` block. The `task` block may be specified multiple times to configure multiple tasks. +A `task` block configures which task to execute in automation. When the task executes can be determined by the `condition` block. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). ```hcl task { From 8457845d2dc526120291242474c20ae3ddd53b8e Mon Sep 17 00:00:00 2001 From: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Date: Fri, 22 Jul 2022 00:18:50 -0700 Subject: [PATCH 31/57] update docs (#13842) * update docs * Update website/content/docs/nia/cli/start.mdx Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/cli/start.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 04534d4a38..4b40b9262e 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -60,3 +60,4 @@ The `start` command supports the following options: | `-inspect` | Optional | boolean | Run CTS in Inspect mode to print the proposed state changes for all tasks, and then exit. No changes are applied in this mode. | false | | `-inspect-task`                             | Optional | string | Run CTS in Inspect mode to print the proposed state changes for the task, and then exit. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. | none | | `-once` | Optional | boolean | Render templates and run tasks once. Does not start the process in long-running mode and disables buffer periods. | false | +| `-reset-store` | Optional | boolean | Use only when running CTS in High Availability (HA) mode. When enabled, this CTS instance overwrites the state storage with its own state whenever it is elected cluster leader. | false | From f5e1bfb2b436db4d58ff952a35786780ca578f84 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 14:11:59 -0700 Subject: [PATCH 32/57] updated reset-store flag and other page content --- website/content/docs/nia/cli/start.mdx | 74 ++++++++++---------------- 1 file changed, 27 insertions(+), 47 deletions(-) diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 4b40b9262e..023e2938a6 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -2,62 +2,42 @@ layout: docs page_title: Start Command description: >- - Consul-Terraform-Sync supports start command for starting CTS as a daemon + Run the Consul-Terraform-Sync start command to start the CTS daemon. --- # start -The `start` command starts Consul-Terraform-Sync (CTS) as a daemon. When CTS runs as a daemon, there is no default configuration to start CTS. You must set a configuration flag `-config-file` or `-config-dir`. For example: +The `start` command starts the Consul-Terraform-Sync (CTS) daemon. + +## Usage ```shell-session -$ consul-terraform-sync start -config-file=config.hcl +$ consul-terraform-sync start -config-file [OPTIONS] ``` -To review a list of available flags, use the `-help` or `-h` flag. - -## Modes - -CTS can be run as a daemon in different modes. - -### Long-running Mode - -Flag: none - -Behavior: This is the default mode in which CTS passes through a once-mode phase and then turns into a long-running process. During the once-mode phase, the daemon will exit with a non-zero status if it encounters an error. After successfully passing through once-mode phase, it will begin a long-running process in which errors are logged and exiting is not expected behavior. When the long-running process begins, CTS daemon starts serving API and command requests. - -### Inspect Mode - -Flag: `-inspect` - -Behavior: CTS will display the proposed state changes for all tasks once and exit. No changes are applied in this mode. On encountering an error before completing, CTS will exit with a non-zero status. - -Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks would update network infrastructure as expected. - ---- - -Flag: `-inspect-task [task-name]` - -Behavior: This has similar behavior as `-inspect` mode for the selected task. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. - -Usage: Useful to debug one or more tasks to confirm configuration is accurate and the selected tasks would update network infrastructure as expected. - -### Once Mode - -Flag: `-once` - -Behavior: CTS will run all tasks once with buffer periods disabled and exit. On encountering an error before completing, CTS will exit with a non-zero status. - -Usage: Intended to be run before long-running mode in order to confirm configuration is accurate and tasks update network infrastructure as expected. +The `-config-file` or `-config-dir` flag is required. Use the flag to specify the [CTS instance configuration file](/docs/nia/configuration) or directory containing several configuration files to start a CTS cluster. ## Options -The `start` command supports the following options: +The following table describes all of the available flags. -| Name | Required | Type | Description |Default | -| --------------- | -------- | ------- | ------------------------------- | ----------------------- | -| `-config-dir `      | Required when`-config-file` is not set | string | A directory to load files for configuring CTS. Configuration files require an .hcl or .json file extension in order to specify their format. This option can be specified multiple times to load different directories. | none | -| `-config-file`                        | Required when `-config-dir` is not set | string | A file to load for configuring CTS. Configuration file requires an .hcl or .json extension in order to specify their format. This option can be specified multiple times to load different configuration files. | none | -| `-inspect` | Optional | boolean | Run CTS in Inspect mode to print the proposed state changes for all tasks, and then exit. No changes are applied in this mode. | false | -| `-inspect-task`                             | Optional | string | Run CTS in Inspect mode to print the proposed state changes for the task, and then exit. The flag can be specified multiple times to inspect multiple tasks. No changes are applied in this mode. | none | -| `-once` | Optional | boolean | Render templates and run tasks once. Does not start the process in long-running mode and disables buffer periods. | false | -| `-reset-store` | Optional | boolean | Use only when running CTS in High Availability (HA) mode. When enabled, this CTS instance overwrites the state storage with its own state whenever it is elected cluster leader. | false | +| Name | Required | Type | Description |Default | +| --- | ---- | ---- | ---- | ---- | +| `-config-dir`| Required when `-config-file` is not set | string | Specifies a directory containing CTS instance configuration files to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one directory of files. | none | +| `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | +| `-inspect` | Optional | boolean | Starts CTS in inspect mode (refer to [Modes](#modes) for additional information). In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | +| `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | +| `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in high availability mode. | false | +| `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | + +## Modes + +By default, CTS starts in long-running mode. The following table describes all available CTS modes. + +| Name | Description |How to start | +| --- | --- | --- | +| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | +| Once mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | +| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| +| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to Run Consul-Terraform-Sync with High Availability for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to Run Consul-Terraform-Sync with High Availability for additional information.

| From 94c7e8741417e473b0941c8909c6727af00bd247 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 15:07:47 -0700 Subject: [PATCH 33/57] partial update to the nav, revisisions to the usage requirements page --- .../docs/nia/installation/requirements.mdx | 129 ----------------- .../content/docs/nia/usage/requirements.mdx | 134 ++++++++++++++++++ .../docs/nia/{installation => usage}/run.mdx | 0 website/data/docs-nav-data.json | 17 ++- 4 files changed, 145 insertions(+), 135 deletions(-) delete mode 100644 website/content/docs/nia/installation/requirements.mdx create mode 100644 website/content/docs/nia/usage/requirements.mdx rename website/content/docs/nia/{installation => usage}/run.mdx (100%) diff --git a/website/content/docs/nia/installation/requirements.mdx b/website/content/docs/nia/installation/requirements.mdx deleted file mode 100644 index 17f13ff5eb..0000000000 --- a/website/content/docs/nia/installation/requirements.mdx +++ /dev/null @@ -1,129 +0,0 @@ ---- -layout: docs -page_title: Requirements -description: >- - Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon. ---- - -# Prerequisites Needed to Run Consul-Terraform-Sync - -The following components are required to run Consul-Terraform-Sync (CTS): - -* A Terraform Provider -* A Terraform Module -* A Consul cluster running outside of the `consul-terraform-sync` daemon - -Practitioners can add support for their network infrastructure through Terraform providers. Once network infrastructure support exists, practitioners can add network integrations in the form of Terraform modules. - -The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). - -## Run a Consul Cluster - -Below are several steps towards a minimum Consul setup required for running CTS. - -### Install Consul - -CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately. - -To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install). - -For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul). - -### Run an Agent - -The Consul agent must be running in order to dynamically update network devices. To run the local Consul agent, you can run Consul in development mode which can be started with `consul agent -dev` for simplicity. For more details on running Consul agent, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). - -When running a Consul agent with CTS in production, we suggest to keep a few considerations in mind. CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, like changes to registered services. This results in multiple long running TCP connections between CTS and the agent to poll changes for each dependency. Monitoring a high number of services may quickly hit the default Consul agent connection limits. - -There are 2 ways to fix this issue. The first and recommended fix is to use HTTP/2 (requires HTTPS) to communicate between Consul-Terraform-Sync and the Consul agent. When using HTTP/2 only a single connection is made and reused for all communications. See the [Consul Configuration section](/docs/nia/configuration#consul) for more. The other option is to configure [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the agent to a reasonable value proportional to the number of services monitored by Consul-Terraform-Sync. - -### Register Services - -CTS monitors Consul catalog for service changes which lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent either by loading a service definition or by HTTP API request. - -If you are running Consul in development mode, below is an example of registering a service by HTTP API request: - -```shell-session -$ echo '{ - "ID": "web", - "Name": "web", - "Address": "10.10.10.10", - "Port": 8000 -}' > payload.json - -$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register -``` - -The above example registers a service named "web" with your Consul agent. This represents a non-existent web service running at 10.10.10.10:8000. Your web service is now available for CTS to consume. You can have CTS monitor the web service to execute a task and update network device(s) by configuring "web" in [`condition "services"`](/docs/nia/configuration#services-condition) of a task block. If the web service has any non-default values, it can also be configured in `condition "services"`. - -For more details on registering a service by HTTP API request, refer to the [register service API docs](/api-docs/agent/service#register-service). - -For more details on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started). - -### Run a Cluster - -The previous steps of installing and running a single Consul agent then registering a single service is sufficient to meaningfully start running CTS. - -If you would like to run a Consul cluster rather than a single agent, refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started). This will walk you through the steps of running multiple Consul agents and then joining them together into a cluster. - -## Network Infrastructure (using a Terraform Provider) - -CTS integrations for the Terraform driver utilizes Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver of CTS inherits the expansive collection of Terraform providers to integrate with, and with release of [Terraform 0.13](https://www.hashicorp.com/blog/announcing-hashicorp-terraform-0-13/), this extends to include providers written by the community too by using [provider source](https://www.hashicorp.com/blog/adding-provider-source-to-hashicorp-terraform/). - -### Finding Terraform Providers - -To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers). - -### How to Create a Provider - -If there is no existing Terraform provider, a new Terraform provider can be [created](https://learn.hashicorp.com/tutorials/terraform/provider-setup) and [published](https://www.terraform.io/docs/registry/providers/publishing.html). The provider can then be used within a network integration task by authoring a compatible Terraform module. - -## Network Integration (using a Terraform Module) - -The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module along with how it is configured within a task determines the condition under which your infrastructure is updated. - -Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. - -Continue to the next page to [get started with configuring CTS and how to use Terraform providers and modules for tasks.](/docs/nia/installation/configure) - -### Partner Terraform Modules - -The modules listed below are available to use and are compatible with CTS. - -#### A10 Networks - -- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia) - -#### Avi Networks - -- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool) - -#### AWS Application Load Balancer (ALB) - -- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia) - -#### Checkpoint - -- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia) - -#### Cisco ACI - -- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia) -- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia) - -#### Citrix ADC - -- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) - -#### F5 - -- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia) - -#### NS1 - -- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) - -#### Palo Alto Networks - -- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia) -- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia) diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx new file mode 100644 index 0000000000..31e8ac186f --- /dev/null +++ b/website/content/docs/nia/usage/requirements.mdx @@ -0,0 +1,134 @@ +--- +layout: docs +page_title: Requirements +description: >- + Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module, and a running Consul cluster outside of the `consul-terraform-sync` daemon. +--- + +# Requirements + +The following components are required to run Consul-Terraform-Sync (CTS): + +* A Terraform Provider +* A Terraform Module +* A Consul cluster running outside of the `consul-terraform-sync` daemon + +You can add support for your network infrastructure through Terraform providers so that you can apply Terraform modules to implement network integrations. + +The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). + +## Run a Consul cluster + +Below are several steps towards a minimum Consul setup required for running CTS. + +### Install Consul + +CTS is a daemon that runs alongside Consul, similar to other Consul ecosystem tools like Consul Template. CTS is not included with the Consul binary and needs to be installed separately. + +To install a local Consul agent, refer to the [Getting Started: Install Consul Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-install). + +For information on compatible Consul versions, refer to the [Consul compatibility matrix](/docs/nia/compatibility#consul). + +### Run an agent + +The Consul agent must be running in order to dynamically update network devices. Refer to the [Consul agent documentation](/docs/agent/index) for information about configuring and starting a Consul agent. For hands-on instructions about running Consul agents, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). + +When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitors a high number of services. + +To avoid reaching the limit prematurely, we recommend using HTTP/2 (requires HTTPS) to communicate between CTS and the Consul agent. When using HTTP/2, CTS establishes a single connection and reuses it for all communication. Refer to the [Consul Configuration section](/docs/nia/configuration#consul) for details. + +Alternatively, you can configure the [`limits.http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) option to set a maximimum number of connections to meet your needs. + +### Register services + +CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. + +The following HTTP API request example registers a service named `web` with your Consul agent: + +```shell-session +$ echo '{ + "ID": "web", + "Name": "web", + "Address": "10.10.10.10", + "Port": 8000 +}' > payload.json + +$ curl --request PUT --data @payload.json http://localhost:8500/v1/agent/service/register +``` + +The example represents a non-existent web service running at `10.10.10.10:8000` that is now available for CTS to consume. + +You can configure CTS to monitor the web service, execute a task, and update network device(s) by configuring `web` in the [`condition "services"`](/docs/nia/configuration#services-condition) task block. If the web service has any non-default values, it can also be configured in `condition "services"`. + +For more details on registering a service using the HTTP API endpoint, refer to the [register service API docs](/api-docs/agent/service#register-service). + +For hands-on instructions on registering a service by loading a service definition, refer to the [Getting Started: Register a Service with Consul Service Discovery Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-service-discovery?in=consul/getting-started). + +### Run a cluster + +For production environments, we recommend operating a Consul cluster rather than a single agent. Refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started) for instructions on starting multiple Consul agents and joining them into a cluster. + +## Network infrastructure (using a Terraform provider) + +CTS integrations for the Terraform driver use Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver for CTS inherits the expansive collection of Terraform providers to integrate with. You can also specify a provider `source` in the [`required_providers` configuration](https://www.terraform.io/language/providers/requirements#requiring-providers) to use providers written by the community (requires Terraform 0.13 or later). + +### Finding Terraform providers + +To find providers for the infrastructure platforms you use, browse the providers section of the [Terraform Registry](https://registry.terraform.io/browse/providers). + +### How to create a provider + +If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registery so that you can use it within a network integration task or createa compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: + +* [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) +* [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). + +## Network integration (using a Terraform module) + +The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the condition under which your infrastructure is updated. + +Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. + +Refer to [Configuration](/docs/nia/installation/configure) for information about configuring CTS and how to use Terraform providers and modules for tasks. + +### Partner Terraform Modules + +The modules listed below are available to use and are compatible with CTS. + +#### A10 Networks + +- Dynamic Load Balancing with Group Member Updates: [Terraform Registry](https://registry.terraform.io/modules/a10networks/service-group-sync-nia/thunder/latest) / [GitHub](https://github.com/a10networks/terraform-thunder-service-group-sync-nia) + +#### Avi Networks + +- Scale Up and Scale Down Pool and Pool Members (Servers): [GitHub](https://github.com/vmware/terraform-provider-avi/tree/20.1.5/modules/nia/pool) + +#### AWS Application Load Balancer (ALB) + +- Create Listener Rule and Target Group for an AWS ALB, Forward Traffic to Consul Ingress Gateway: [Terraform Registry](https://registry.terraform.io/modules/aws-quickstart/cts-alb_listener-nia/hashicorp/latest) / [GitHub](https://github.com/aws-quickstart/terraform-hashicorp-cts-alb_listener-nia) + +#### Checkpoint + +- Dynamic Firewalling with Address Object Updates: [Terraform Registry](https://registry.terraform.io/modules/CheckPointSW/dynobj-nia/checkpoint/latest) / [GitHub](https://github.com/CheckPointSW/terraform-checkpoint-dynobj-nia) + +#### Cisco ACI + +- Policy Based Redirection: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/autoscaling-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-autoscaling-nia) +- Create and Update Cisco ACI Endpoint Security Groups: [Terraform Registry](https://registry.terraform.io/modules/CiscoDevNet/esg-nia/aci/latest) / [GitHub](https://github.com/CiscoDevNet/terraform-aci-esg-nia) + +#### Citrix ADC + +- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) + +#### F5 + +- Dynamic Load Balancing with Pool Member Updates: [Terraform Registry](https://registry.terraform.io/modules/f5devcentral/app-consul-sync-nia/bigip/latest) / [GitHub](https://github.com/f5devcentral/terraform-bigip-app-consul-sync-nia) + +#### NS1 + +- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) + +#### Palo Alto Networks + +- Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-dag-nia) +- Address Group and Dynamic Address Group (DAG) Tags: [Terraform Registry](https://registry.terraform.io/modules/PaloAltoNetworks/ag-dag-nia/panos/latest) / [GitHub](https://github.com/PaloAltoNetworks/terraform-panos-ag-dag-nia) diff --git a/website/content/docs/nia/installation/run.mdx b/website/content/docs/nia/usage/run.mdx similarity index 100% rename from website/content/docs/nia/installation/run.mdx rename to website/content/docs/nia/usage/run.mdx diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index b89ec40984..1d7b1c3000 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -794,16 +794,21 @@ "path": "nia/installation/install" }, { - "title": "Requirements", - "path": "nia/installation/requirements" - }, - { - "title": "Configure", + "title": "Configuration", "path": "nia/installation/configure" + } + ] + }, + { + "title": "Usage", + "routes": [ + { + "title": "Requirements", + "path": "nia/usage/requirements" }, { "title": "Run Consul-Terraform-Sync", - "path": "nia/installation/run" + "path": "nia/usage/run" } ] }, From 4e2d32ca804dce1c010d422333d3912e1a374e8c Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 16:44:14 -0700 Subject: [PATCH 34/57] add HA usage page and small revisions to standard usage page --- website/content/docs/nia/cli/start.mdx | 4 +- website/content/docs/nia/usage/run-ha.mdx | 184 ++++++++++++++++++++++ website/content/docs/nia/usage/run.mdx | 13 +- website/data/docs-nav-data.json | 4 + website/public/img/nia/cts-ha-after.png | Bin 0 -> 28850 bytes website/public/img/nia/cts-ha-before.png | Bin 0 -> 29219 bytes 6 files changed, 200 insertions(+), 5 deletions(-) create mode 100644 website/content/docs/nia/usage/run-ha.mdx create mode 100644 website/public/img/nia/cts-ha-after.png create mode 100644 website/public/img/nia/cts-ha-before.png diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 023e2938a6..801b14ab6f 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -28,7 +28,7 @@ The following table describes all of the available flags. | `-inspect` | Optional | boolean | Starts CTS in inspect mode (refer to [Modes](#modes) for additional information). In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | | `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | -| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in high availability mode. | false | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | | `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | ## Modes @@ -40,4 +40,4 @@ By default, CTS starts in long-running mode. The following table describes all a | Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | | Once mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | | Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| -| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to Run Consul-Terraform-Sync with High Availability for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to Run Consul-Terraform-Sync with High Availability for additional information.

| +| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx new file mode 100644 index 0000000000..50e8dda098 --- /dev/null +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -0,0 +1,184 @@ +--- +layout: docs +page_title: Run Consul-Terraform-Sync with High Availability +description: >- + Improve network automation resiliency by enabling high availability for Consul-Terraform-Sync. HA enables persistent task and event data so that CTS functions as expected during a failover event. +--- + +# Run Consul-Terraform-Sync with High Availability + +This topic describes how to run Consul-Terraform-Sync (CTS) configured for high availability. High availability is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. + +## Introduction + +A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process if it is configured for high availability: + +1. The CTS cluster promotes a new leader from the pool of followers in the network. +1. The new leader begins running all existing tasks in `once-mode` in order to process changes that occurred during the failover transition period. In this mode, CTS runs all existing tasks one time. +1. The new leader logs any errors that occur during `once-mode` operation and the new leader continues to monitor Consul for changes. + +In a standard configuration, CTS exits if errors occur when the CTS instance runs tasks in `once-mode`. In a high availability configuration, CTS logs the errors and continues to operate without interruption. + +The following diagram shows operating state when high availability is enabled: + +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.png) + +The following diagram shows the CTS cluster state after the leader stops. CTS Instance B becomes the leader responsible for monitoring and running tasks. + +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.png) + +### Failover details + +* The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. +* If failover occurs during task execution, a new leader is elected and will attempt to run all tasks once before continuing to monitor for changes. +* If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. +* If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. +* If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. + +Note that driver behavior is consistent whether or not CTS is running in high availability mode. + +## Requirements + +* Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. +* CTS Enterprise 0.7 or later +* Terraform CLI 0.13 or later +* All instances in a cluster must be in the same datacenter. + +You must configure appropriate ACL permissions for your cluster. Refer to [ACL permissions](#) for details. + +It’s not required, but we recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. + +## Configuration + +Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](#) for details about the configuration fields for the `high_availability` block. + +The following example configures high availability functionality for a cluster named `cts-cluster`: + + + +```hcl +high_availability { + cluster { + name = "cts-cluster" + storage "consul" { + parent_path = "cts" + namespace = "ns" + session_ttl = "30s" + } + } + + instance { + address = "cts-01.example.com" + } +} +``` + + +### ACL permissions + +The `session` and `keys` resources in your Consul environment must have `write` permissions. Refer to the [ACL documentation](/docs/security/acl) for details on how to define ACL policies. + +If the `high_availability.cluster.storage.namespace` field is configured, then your ACL policy must also enable `write` permissions for the `namespace` resource. + +## Start a new CTS cluster + +We recommend deploying a cluster that includes three CTS instances. This is so that the cluster has one leader and two followers. + +1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/installation/configure) for all configuration options. +1. Issue the startup command and pass the configuration file. Refer to the [`start` command reference](/docs/nia/cli/start#modes) for additional information about CTS startup modes. + ```shell-session + $ consul-terraform-sync start -config-file ha-config.hcl + ``` +1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. + + ```shell-session + $ curl localhost:/status/tasks + ``` + +## Modify an instance configuration + +You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings, when high availability is enabled. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). + +1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: + ```shell-session + [INFO] ha: acquired leadership lock: id= + ``` +1. Stop one of the follower CTS instances and apply the new configuration. +1. Restart the follower instance. +1. Repeat steps 2 and 3 for other follower instances in your cluster. +1. Stop the leader instance. One of the follower instances becomes the leader. +1. Apply the new configuration to the former leader instance and restart it. + +## Modify tasks + +When high availability is enabled, CTS persists task and event data (refer to State storage and persistence for additional information). You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes. This is to limit inconsistencies between the state and the configuration that can occur when mixing methods. + +### Delete and recreate the task (recommended) + +Use the CTS API to identify the CTS leader instance and delete and replace a task. + +1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: + + ```shell-session + [INFO] ha: acquired leadership lock: id= + ``` +1. Send a `DELETE` call to the [`/task/` endpoint](/docs/nia/api/tasks#delete-task) to delete the task. In the following example, the leader instance is at `localhost:8558`: + + ```shell-session + $ curl --request DELETE localhost:8558/v1/tasks/task_a + ``` + + You can also use the [`task delete` command](/docs/nia/cli/task#task-delete) to complete this step. + +1. Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. + ```shell-session + $curl --header "Content-Type: application/json" \ + --request POST \ + --data @payload.json \ + localhost:8558/v1/tasks +``` + You can also use the [`task-create` command](/docs/nia/cli/task#task-create) to complete this step. + +Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. + +```shell-session +$curl --header "Content-Type: application/json" \ +--request POST \ +--data @payload.json \ + localhost:8558/v1/tasks +``` + +You can also use the `task-create` command to complete this step. + +### Discard data with the `-reset-storage` flag + +You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/options) to discard persisted data if you need to update a task. + +1. Stop a follower instance. +1. Update the instance’s task configuration. +1. Restart the instance and include the `-reset-storage` flag. +1. Stop all other instances so that the updated instance becomes the leader. +1. Start all other instances again. +1. Restart the instance you restarted in step 3 without the `-reset-storage` flag so that it starts up with the current state. If you continue to run an instance with the `-reset-storage` flag enabled, then CTS will reset the state data whenever the instance becomes the leader. + +## Troubleshooting + +Use the following troubleshooting procedure if a previous leader had been running a task successfully but the new leader logs an error after a failover: + +1. Check the logs printed to the console for errors. Refer to the [`syslog` configuration](/docs/nia/configuration#syslog) for information on how to locate the logs. In the following example output, CTS reported a `401: Bad credentials` error: + ```shell-session + 2022-08-23T09:25:09.501-0700 [ERROR] tasksmanager: error applying task: task_name=config-task + error= + | error tf-apply for 'config-task': exit status 1 + | + | Error: GET https://api.github.com/user: 401 Bad credentials [] + | + | with module.config-task.provider["registry.terraform.io/integrations/github"], + | on .terraform/modules/config-task/main.tf line 11, in provider "github": + | 11: provider "github" { + | + ``` +1. Check for differences between the previous leader and new leader, such as differences in configurations, environment variables, and local resources. +1. Start a new instance with the fix that resolves the issue. +1. Tear down the leader instance that has the issue and any other instances that may have the same issue. +1. Restart the instance(s) to implement the fix. diff --git a/website/content/docs/nia/usage/run.mdx b/website/content/docs/nia/usage/run.mdx index d4746eb03d..a90cfae1af 100644 --- a/website/content/docs/nia/usage/run.mdx +++ b/website/content/docs/nia/usage/run.mdx @@ -7,13 +7,15 @@ description: >- # Run Consul-Terraform-Sync +This topic describes the basic procedure for running Consul-Terraform-Sync (CTS). Verify that you have met the [basic requirements](/docs/nia/usage/requirements) before attempting to run CTS. + 1. Move the `consul-terraform-sync` binary to a location available on your `PATH`. ```shell-session $ mv ~/Downloads/consul-terraform-sync /usr/local/bin/consul-terraform-sync ``` -2. Create the config.hcl file, all the options are available [here](/docs/nia/configuration). +2. Create the config.hcl file and configure the options for your use case. Refer to the [configuration reference](/docs/nia/configuration) for details about all CTS configurations. 3. Run Consul-Terraform-Sync (CTS). @@ -29,6 +31,11 @@ description: >- ## Other Run modes -CTS allows you to inspect your configuration before applying any change and to run in once mode, meaning that you can verify the changes are correctly applied in a test run before running it in unsupervised daemon mode. +You can [configure CTS for high availability](/docs/nia/usage/run-ha), which is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. + +You can start CTS in [inspect mode](/docs/nia/cli/start#modes) to review and test your configuration before applying any changes. Inspect mode allows you to verify that the changes work as expected before running them in an unsupervised daemon mode. + +For hands-on instructions on using inspect mode, refer to the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. + + -To learn more on these options check the [Consul-Terraform-Sync Run Modes and Status Inspection](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-run-and-inspect?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 1d7b1c3000..224d5184f7 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -809,6 +809,10 @@ { "title": "Run Consul-Terraform-Sync", "path": "nia/usage/run" + }, + { + "title": "Run Consul-Terraform-Sync with High Availability", + "path": "nia/usage/run-ha" } ] }, diff --git a/website/public/img/nia/cts-ha-after.png b/website/public/img/nia/cts-ha-after.png new file mode 100644 index 0000000000000000000000000000000000000000..5cd256056a5de45ae101325cec97729dbad39de4 GIT binary patch literal 28850 zcmZsD1yqz>7w#~0r*wmaG|~-{k^;ieNOyN5-5t`3ba!`ygd&Y}Nq5J6@vHx?du3b> z@0>Zg&#vd$hAAmXp&}6?fj}Ts8EJ845D4rC0zru&zyVJfM@OGOfO1rp5(AZw5&s5$ zQ8I?en8?e6UIFhBKwu~=5cG2s;0F|94ubjf9t5HR{sn=cbD;kFGza|8r(nMv=zrcr zi99#-9`i^9I?*s!g*ZXv<@gM3t(iX<*%}x#yII>kHvkE^@d0nGjh#M_yIEV=IP$p( zQvPYd2fTlN%tA^2r-_rLASFaziTsVNgE2W5Gb=MIr4SN1Ik|v?kqMu&xa8ko2YwTz zG;?yY<6~iQb#-NSea&p^V9LVA%gf8c%Fe>h&IGhza&)(G`ryW7<4E;aC;#*#ZtQ62 zU~cDRZfisS-0ueiTW2RhO3LSf{{8dUJe|x<{u{~0@$Y2;3uJlzgoTZnmF3^Qfv*ZY zKjl+&FgFH9e(qn0P2f+<|LfV`;|Q=k5B@)s`D>+ro&rl1LK0y4ciV)J2y|EBK_C&3 zjJT+(8`OR#te2`3UY{ZimKXx!;i2xL4pQG+dK}>A5GVXaK~S=C1};-jeWOZFqgLD+ zj4LgVf9&xnNxM9D^~vbg!4^u#=;0XA!?@wqeBG~4$eXvA31KQECCiWPiQkKyR} ze;Q*t(ek&Fn_2k!O$-e z4EdBm{T!>($z;d3s9;jz$)=_7x}!R>b4Tit4-V@Xc9+tHrT(=rgr?%NSkLQc<${90 z$ow3nkHVE;2{Od6Neov5E5T#>R?t0u>7CI31lDcif<(*`|Fv0E*ls1YegX#Uz%bOm z3KN+upAs*zzvtv{=PHBG@FPV!iK276f&)kY%&iC8ZLi?muk<12vj`Oa8~t3BeJx>) zBfLzk)Rcctr*j$+rqEcQ?FAjkFx*OYWGY?AV0xcuDxZz)pN;l=fjH>EItN1t?GI*! z%aJUQKG=Q92g}OxZ-ZYL2sxnI^eU7VDqBOFcH#YS_oROZu!H${4P9@vUlHgqe_!o0TTS-Z1w$r{a5k>k6gIF?qmG3W)NbaydRj=N~rbifBm~i`GF@6gA++8 z|B-BR|4KkigGWQuW&RcB&O-zx(EP91>No!_3WEy}@5-CwSO4`tO9F)?(n6%{68g_p z`oVP500+rMru*{W{-Ii^R;*kF4gSo(a|}_y^Msh2Y_Vsd!4`E!MM_HQt3o{Sp^Z(j zB@q!08`#yGbmeTfG?6#=+ZtP4L>!xl!bg64-~JXk<>zc z*~z=IG`e@KbZQm4)*)@iDM0%rV4!Ou%=W+9(;VMkSl-{CacqnwVXyZ`_4xTi+iwnY zx@GgnyiF>}>DwMv%9o?Jnyp-lrmwUFwi-~f@Xs@j8yh*2at90>|TW zKWx%;?8}>w=uk3Yp?F$Fe>TJR8T-D77gFnz*I>x60iA6s9uLb5ilkNdG}FnjF4CqA*WI)gbhViZ7U3$meO=A z1Lq?qlvy4Jj-MKM)Z)(?>JcLGg;2J~GAEN$EvAa8K6D4Ja=rAvJ&&zDZ8aGsWYlgX zUlfqTXh4K{;nNp^7jl1$&u)Z3$Y~Mi`}BZkI1rtO!7;%k24$AhrLXd*OBw|2b}xd1 z`MMA(1N1~j4PyB;j{YK*09p#JzoNilu}X;?zD($2DLg(q zKSxP!6%Bx+2wZ7(Kl_-sO0@T^BuFi%|EQ8Gv6X{`hQZJI%eah?z&6jT^^1}jI<;C> z&H8)drKKf*>-k!9s^Jud2%2n8$vJjo`>j&$_B$I60YPl|{8eNux9m>gr~BOLa2tsx zht1l$Bk$EKpWEH3lJ%N#p`Iur?nDL-Q&ul7_Fpb`8*dEzcHkhBNdf4#A;UCj2~P70`uPYxZ4C@ycW*S-#@N za=w;cAzO&#%hY(5po%bE`>K5&F9_ogNYQ+NuB@Q z;xWn(KcQD^<-b4YxH5Z^Fqq1g8<%FTVtpd$4%<#!9z1U{<3+HN#!ao4yXh!Riawah z|I;V6C6?Xz?Sb0PUPVX$&iC9pj8d%z=Jt0)F-9c6H--`}2t{uEtGsR;teH5}(bh|! z)uuk@FzS!xRGrLm0qDY ztE=eriB6*()URge0EDh=pIgL-+Y4lzj|vJ3niy~|e5x5goUHZDRvSm0%HGs884%$0 z0Jg&rpEcsot;4D$1JlHrDtrs0OT3==sLhQRnS4|Z0a0PTO?nk%Jd#Y|xbD{8l)knpJ@cL8?_21&FEDg0Qg)(lhpy3PGeQW2Hg{0EEy ziBTvC&|OteX5HU*%1;L))-i9BgMjy>Cj;YK5<^NB+U3<1I>o(is~Z$IH+Rd5zGdbL zv%VG)Tm2u)PANlfT%0)DV9R3k>%CFW`aq2RS}$T96_12b4ZAln1-a#8FA|UAmYiUj zCDuoN*TbJJ{sWN&0Z`CzX3Ghq>D)7F9S?U`xxgjC=hI(&U2$%IK&3KS%w ziFsKK`y=(mjsX#1MeOjtyZmf2nx=W5#H>T;2`DYkPLv;pfGdCklm15|vq|JgfB!NZ z?uzxkSZ#7FTQ-F`>|$@`Q`zBEiFzu#F}uU|aF|R2&02a_K$YVRDYb%}>4b?tn=&>= z3QGt&*|eA)a22}(bEsJng?ZV&%6(#=w_V8*4LFZMri}dGP4;__NG+SJUDiVw@T@si zhm@~N?Wt7h^c0#_XG<4O0Of4l^eMX4-C@;})A;Irwh)i=ZK;r$!n7+}tcWmZn465nGifFbvuWpc+~IaSaH$_nBz9xbUWy+rwSE6=NJ}jpORl-8*X%~h z``Rd4jX?4naWa_*RXz7B>-akcS85W`5Vi5)B!&z7fP;mSn!+h%uq+I(aWCmnS>-HG za}x*+?ZeBkb55jAX{v}h6dXUc5pK1&1)#_m`#vQRV9sT~*#eRkH0MZPmlKrFI zOE8a}6a!1tnF*#5TV%cPzRIL6S_nYmj1dPPHV+Q02k&Hivuk50Zh=kB9glqp3GJ-2 zoQdTyS8i--(kK+ieFMIfArDel9t`k@G582Nsz}YTXl3VxwQ6Mxb7a!2GJr+GWei0j zU?-P}r+(IHHoux*Z1Yt102|g%yOPEN-Ab4gAn-cYj|p&+Yd62woYX&E%;?c+*3zYM zn13vOU-IntL8^@qFAsl3Fltn{w>L)kF*K6k^%ubLxZck8JqG?70A9~v2Jh?J ztD_j;?#`Rp&RPBK__2}#?%aA**suDXtum16U}v?|qZQbz8gSIG~Q9zgiVee@p$Ru(DndMlLDe=L*VaG~DXY`I;qKZY#k z{P$#tM*RMz#m_=aUaB( zWy^k0&=ci4{Edx`sb#CZVc7V5PISpkTKNSL#ukuaI_0mSbLWxNP*hkvg+8@%8GHl< z3fW?7y;;{gJ!w1BMXyN(eIhI1QCSx@zFNHgiM|}&pY?FZ`r(v1gU21q{Kr?(sN$)K zHb=bP2s{=@`-pnE{t;l?xRxA4Hhc+l< z*(AxmWW8Q99macMaLx8(xQ^%5Js#%9`;o2eT(LO^QG;&ZCaf=Cre$8>A1yegV}dVK^?E_;}1B>dR9TmcwD7t3xz5l6g+7vEk-RkZd*UO zXltDZbRP?CM;-9H>Q7EHrE(n7e#;adx4L>ww|7l@L6mbEG8NZ5f=s4X;h+2TSOU9# zl0eJq+Fz=Fb13YVGvaZ6{jL^9zwFleWaBrswtzDiC~pP#SDcQ*zVxg83XgxXQQe$ zvzh!dpHbmts!n?Skt!n@Olor64pKpnol!Et*_Z^g0&c7(PQBF}miyUOP)tnB5^Af* zg$2_iC4?g(&hG<)fbS!*HNW$Ou+^o_LRe;cgT;sUbq+7?+st#@pB}Q?0tK2~_DdJq z&SQdZAMz%bAk78@_xJa#-lwYH9^LuS_7|h#`yvdNA0OslVW~;=?YfC0_LmGdY(>u1 zR9bB!=_J#SPaiC>I^T|`UJV>I`m6^>3*|8GkLwA9R8&;p7M(H{cvz_!I1jT-o~JY* zupzSOwoaH+AB_pNIFoq3Z}}K}hR346l`(gHaCL(*VlZ(WbNi0=0IMVx;vt4?0F}fy z`D$-Fv&FRz0fnioZcXC;hH<_~BByRQGRnh5{!U^!0^N3LdTM(#Q>r87u5J{s%x>+c zj9@)7E=zY(i8^#T-PNex+t#7Rk(kU0h0>d=Yc}s++uw6LTSfK0eYzsj*3nsw8-#ZB z<$iiYZ|2b+E#N=yV`q9e^+ZQ$O&q2BYoZL} zx$L1GR9g4YobN0USthp7RCQ{%`Ji(Dlq`2TXBOQcUEItEX~UBZU`he9~&L-Lp@uVyVavib$r< zTSwX$33+%T7SBU^aUqCZn@35gRcl?w)I2#7cwd^r&TVy1t(YK)XbJIWwjffF!e5;2+0x|qrWvd>JOb7cMT&Si~?7<(sqD(vDpds_heqr zZk?smj|<~Ne+bRp`A!h6BB|j>Gir00fpv@BDN1XrK-?@9o6qedyw!3e z!93;HX~VF-aU1~wtkuVh^w~z2%zE5%Wy{vy)44izMI2TBhl*!CqMG&@~7|Bic|^mWD^7Bki!d2LIuGF zNw|A@fiMQYyKQjhHY97h@KnfESWeR;WAq& z<}{J}1Sd5R%P#ICBK>KUkh{wiRd_1%MFeGlF=8OK?cAvb;dAuTD+AWe(rsQwClx}N z_MNMw&@*9|AX1BsB2^hs?EO_TQ?)-X;Q<{O+PqcP;U`xSj3<6>Jf4P%h(#@e_EIdH z{sYbDg!!PS>mh8Tp#<#BV$~mdV>f4;=e$}Q zn^)U~_nq_>sm$L(3e>CxgJ>=BV@L!7m$$$5l?%2wO=p`bNKLmk)g~!XUXmgRc{AFx zTQlup0zP@J`%A?*;tJii?vH)Qo9u5GF=J0ipL!snoOcQNQ71A9lFcunplm7BgsYH- zfBK%Do3r|MWf$f$)>jt|;j};aMB=*E{cKh{IoD?%oQKfT(k{06Dr)5!|Jpt(E}h6v z7)|PGJbuoAjfc`ddt>B3V2t?W+Ix-gN+5l5s{@GZ-`~oA&k}N+$1%Qa%Ab#oW74y+ z)+kXgh^|j|vY64W*yMLVO`H=DrCR5GYEYRqGR8c){2o$gx3yzms#}fnQZ})3BfH?s z(~78Q&F;)LbB5H#eBFxoBXKY~LAb=(6GB{nEzz86J2mUUZi>073Pu;Vbw7)p?D1iS z?OYM##OgRnw=ziG+YWT=^~m-?VK#-8a=y$Wxx8=>l0Cdp zp#j_1rts7BlXV6!1R5<_;LZ==ZEbD6=1URLNO-YQw??L|V_zA+b=qHgu{Hg4z;6dC z7|;5?r_JxQ_wiDP-?EEtPu^JYRlw$Ra3B3r_BmDPF9K#yny)jY4ehIN%0t3ucWBhT zqBfw5;9Jf2$!zvFA8n&M?SgM~x?$htyoYi0NiPXX3zgBp*Av&!_=45Drkw8p`_uL( zhWpt@kQae$>~Ai_u`vVZrV3ZREFo{rf$fn5DMi!@HA|?MFm<5a(5uG{_4G!gd9Bin zY6AV~3de~uR~BoTQjPUD)^UmQti$)Y9XloTaW_X*hFx#td`95F8p0;?bX4P0#i~;V zV)^6*1|fb{*KU{h1aDIqqxdYMvaCqs$*+PKU)wn-=Jb6yIuty%Z}Z@LEUMl}6I)=> zG~9PJJnzrR^6p@lfERk=d&)f4J6`Z=!x2rUV!Sc*Z-2C;YuSQ+eJ1f0*cnMAJZz~D zztW4AFYPF~SVpFEB%BW~jag&;$}lkWe5gWDR3bKx^fz{n&N-4xpYA*_^kU_e^2sN^ zt`UunGWpZLtw=~rW+RW<_y!L-5wc#H!V;%Ad>z*)i6Qh~<>J5u zlcp24#@OLeN&n9Y^7UFzD2Q;5bLXY+xHPzfX<3 za)*Anhbp31$VMb!klM~=vj)>Pt$28o^I0ahKNeNA8rl1x*;SZv<8~y4rAj&zK3g&q zZ7N6CTX5PzZ>mH`YM@MYdS$?(?Nxv=0uSBPk(U+3O57jnfjhRN5X!(Ct|Nz(b5!)YHuk zriFB#s7l|+Sr%*Fc(0G2w`Y#w^Tn>Ad@waRKasJAYOk)*$5Kx$S7u6l6>3G&Po*F7 zlP&@cv|&0I~ToeWBC zq1}Yv%SEL)H$Wpeyfs2omX4P#!AzWr_^E6EgY z9Sg{n9?s4hj})s{Lp{L?Fj~>5PaO5sn~(8#IyK5Egy&7zX9(>Q*!T0*wD>T@rT)QB2FC`VNdW82)bSeJpQ?2ve zX~F!Gh`Q4zWS;4Xf#2bVVju@>>5TTgS$80G(|T!lHqB)9UT9XRT#E&Qm-S1f{A=Rm zMnXu;vp+LGllB`2RfbL>v0ywcOc<9Lp%y><%+$WsOTMfxiJVzzqwP0}I&TbQDZ&%o zJvn!$S9tZ=g*G2C$&_;89cgOoczl9ECovE4^-|Oh8vl|RYYK-|X$%BA^#UGN{D`q4 z880gxub92LEKa^ijIv{#aPL1!m6qLFMk$>2D6T72y1hFnY%pvnAB+=bsXv-83ewq3 z<2*Ni8Lu>EnRJzJIMaN-4`si5zZKOHU2!J^ty-uH`|IvpGOD?*PDOu6z17SEQBtm%3-t=h75Lq*S(!H4cfYG3 z_pk{^Pf7WBtJWm9Gy85Ju&Tr+ypq_wAaUC+|46AZ9>G;(*QEl0 zUXcHjPcuOdo7Eq*D^{czA(25vjaILdpAh!ubpG;g83po#` zL+TbI zH*VYRhhl}=_-441d|3DcHY1mFK60>9=sjoG!$A)qLJJ;{)+w~xNglex#^7f>e)?d1@B-_e#wj-P$Oc627*Z1d*q)-sCQxeA0$Y6c{imA|`e?r+d^wo6626N0X z0#WD8Bv1ff8bGTjz-E=0wJ(u#x+Vx3cxyp~u2xp9#?{}~fsn)EEtkd$P0Z}>j|X`4 zmF;bK4O0M#qSn^{vweX23aZ`5muTTY>NKO`(1pVB>nFml|AVbsg^I`_h}eLuJ`?%L z{|~e(3#9aFVI-3O2Oi`S2QFtYT*TkaLUq8^$k*Sy{~x6RTwNfie4W*Zlr8ZVRQ#7p zae_7+$u;e(Usm_&NiNfhYPmY3(D-3gsFzEloF{Gm9hXt3MUzD41zBCmRHa!{!_iV# zf967*SY3-`-!LvydeC3$9!SvvR;RY_U5!I}OqM`&+Ne-pk5mlCI4|k_()EW<*Pbe` zX#rKb_x%QCzxf>y;@vznv_T4){6vQNa(?N2h^xb0hR1e_Mpaxj2UQZxafIJ4$+?A&UX|G=!f-gGn~^by}h`~ zdFQsW9x!nlW~^8q@}gI9{@ysE*3SF$7@JNtjRP%B;Crr$eNkPOqZvk!k7Cu?Zaaw-*~N8K#bB=gbCl$_KBg_;+Esr8**;;lzWgn zLsEa2Z_JJ*9V`i(RIudT1Ut$%Pndjft&ypEAQX=jD zvC9R3KiXfk8m(!1=duf@7}JZ-b?C`-9PR}{)N(lzIo#)U@*hp?Z1Gs<<6_eE4-y{l zp8m!E-S%f?=dfDo6EZV%IsVT!Zo;Xeigop;HeKP0xmqZV*T#~o_pso}kqF&4@CH5g zOsO~P;#NAt;CTPPQSCK$05fkCxW(M}{Qfb>&f-o;j*t@$)ncmuh4cB(LPX)U+^wj( zW9$+Aw8WxH?{Ve~9!J8VDU*m7doys!SGRgdq|WK zovZ$SLQ)HE#bo&MQ-xJgmx*}f1xwyYR)f>K(Avo7wE1)J(bY8&-Q+KhhE3k^{UfBoP);Bxvwsr+c_iA5Q#*qHn^mkwVe#NTpMm# zy~1vz*M5Uei}Co>>m>;OY9D(xzZ2Q(&uw4Dup3Q*(xroUHvwD_fOTO8+}xo0hzUNs zbJnpE^GN|-hc7NugpFPYud)O@i*@R8n69DXX=TD^d@8=Tyuo8C4vr!UP#qRCMH1~x z>-E-b5%PJoM)OJKLZn1H+Z?ujTbylNP$h?W#Oo$v3zAt&5<*m& za^;+I==Ly=9x&h&vN$K!{Po6q2;6q_hB0Mq?8xQzXoi4))D+hO%W>9q9Z%aR7 zx$G}Td>wAkTgqqxPYxr0;%l$a?%xryU7Q&1#MqNO&U*?UL8s|3>#iI0k69KcrhTg3v1 zYsV`!dw+Z;Z!jAL=t--I?BTKs=5NjVN6Lu+49N^SpqC~dI2HuAyRlD=Wmf?QG5AoP zq_#kA`q*q$Ves{veT>TOyOA{U+&aXKq2@Pk^#`dAizcHqQXS=PzY{1lLelI)@yZUR zIw%Fg?UrOG*s}VodAgB;Gv`|{c2PmW`(fVhT+a#L^f@5ljRoySVr&!ZeKRy!h(AL zg1MyyQML^EE0~Xb>(Or?Q@>FqztVhSl${-1XKLAp2J;-)&S5s#m`k>IT)P{5@}w9B zPq0tb;Qa3%TRzJ>m;JQVKLM2pxC+4OSd-!Xrv5rb!2gN$s{^u8n&~`yfKT?10)o(B zZ0dAYE!wCl{{3CGXGE-rACL9_0}ATD2bT|DW&h=k*dfhMOT`}i4-P1e#hL*-@m&xq zBcZWpXi_JW^%U2~m8@%dn_#Kk2YaZTjTdn1RS_3kT+zd@UVglGH2iBGmBQd)48N1t zd4ZmN_jW7^@D6edccK|wBfm+;U9 z$dgZ0=m`S|_YbVT8k7LcMvseElF`Ir5)rsvJw4Dsa#w7oTqln8gIGXa;xjw2lEWX8 z33gFf4a4yk7QHpkT)EJ2uUS*%+h4 M{rcaQNHJhoJ%E>{tgIF)-RyNtrfa)>os zZ%s6u!V)uIXPK+t^^wQ@lp2>=8zF2w{v~qz!BvzXY+r4p zqF@exLvRY6>ui}c7d2x#?eNW~I2gbPYpkmSn56ZEY$hKkz`BXjs4<~X$(N(3^g11) z<#E}^_qo}`13*-9sW4QTd3l#0I)=$)nGF<{R?1bt@NM^qfGBypRDhh zuK?iR7u)4_4-X!#dMik!zG$VB{Y}(SSjn-*SC||R6IR7e>U*P05fdx`s0sv<9=X;F z^*A55;`-i}PQm7c)*BuC42k|kuRXVa;x-DkcxMi4bD0^i(DT$A@QLVnE!)xKjkJ=O z&{@OU(0qVs?sXcFul)$)alYfvX*s=vz5xL8V(l9}NDi!C2{iJxAV84Vy&HX>9xJPl z)BF)42|1BYW9c7zLeMSuY#^hK_V)gZR8Nn06}srNwdN}Oi%qXf8Cxa69&>(>d%5nT z89m>iZ|neTLW_md&cWdeK;iN>cn?KKB%tgyMILyK>~@KDRZ!ZuX<}&adrBxUXLtQ8 zKSr?N#B9iBk&pvjeL78^|GuBfM$Awa1Y6)Altk_`ZmFmfI% zyl;OOc<@P12cnYj;qW?a_(7nJn@vQ>< zva`v6tQ_2@`7(`~eB3iKuVepuF#LhPS6n(H&uNg5K`(-vr>7+a{&_n>T1bVA{`t=L zou#%GUByeW-~P6Va9mW1yJhpb^Fe=z&E0Ax0A}Y02bXCw+Jxne-{~FnNZ* zX4KB0ci!dNg=YOq2Q1d15E>@gr_ce0tM!8tZnx7J;kcpxT41k55ki-_Pp)WvRshVu zV#YV@TC2^&IkjUe00;AMq($$E&?XIEHfNf(*+?EWEB5v0S%n_3|Q(sCE)NiO;k=KCu>cK(cx04dLk5IU)`ADpgfmWDy^PGJvX z@!7J<-YpAmc}?~^274ZSv6I7z&nw$No}a1I@CTWf3gtK=DjR6l1S)Tz-`HA<<<^44|_^l_IA6kNO{Zrek_ z{vLO&FY!t4Kn7`vihRs(bEdTvbMHr9L|<>V>e=o@-<3tp;etB+xK=}VYH67PLquX0 zm$hcwqsn53YLT*pRsfT7<*VM4d!1`59?UO1EQzmKKu;YlT;jn`Qur^ki@ed^t_VKe z(UFeNn}s8tFbeZvc`LsXZw1!!{aA^zSGE%X{*{`Yev42vx}6xL2IgF+{UdaJ;=BJvw-Hj^UT9lcQ+-^wf$PEAoEuT3(Bj^BQ)cPb(AJ5MpHKTrLdBHB!E->sNTIGAQQr!5I( z23P#zODQENGr>J@=dGCn$EFd6JrU0!rAm7xoH^Ap_95N3%GjT zCT<*PI{JZwW-Fmu>QP6kSJw53$E6v(-8jaYvx`6N++cfMgkosoItq+5XK_!129}(3 zyu!nCO;2dZCp)V>wF7g^GmGedm1#Awnta1&y6?SdEQL=TH3dk0ov+eGG7_kfzVknY zD8A|CZKf6J#_Hk@A7ryT+&m4E-g|9ZGLI=h7NdSJ5D~C6FaNC#k>Sl(WTv&U4mMZT zcJK8VyuFyfX{-Lf{VTs!gh4(74V717N-k+5X`R4s?s5785|C0U2P^D$3|bR=>jF}_ z?>l7GlmM+AEMT;G?UAuFb8xeo^rclN@(!(O7P#uzDX+1yBe#+~X0wR*81RZQ`q_{b z7s0||g^Gd?4-Xq%4m1-jFkH#2U8IKfD^bDK3;cXi*D|{HXrm(FErdaOF@T^u(=rl( zB+6?p3CLmGGY=)_I04lpe^N~7ZvVBT1p62gOes6;eSc(~V!qY#qBl-y1Qd(z>f9x^ zT}7%etI$eQML0j{_0l<|%vrd!b_Y&|3E-?D$Z<9zW?Az`^XKBF#|PeIIN2FCs=vZ8 zNtL_G>kx8pQ2jE=XBCwqh(OSC3)5sHAc<8d?LfIcs?$zWgW|u2k;`6&O>HWX=0j`vuUQ>a~Ih@{EN?~<%`zKy>h>PS9n=QA;x6aIW$S1d_BsE4x z*h6JqXHyp~$1Og3wm9z9kFj&L0K*CkK}ICsRD5s<9-wUQUBK7yfRc*{Rc7lf@m{?z zhLK@F)7?}>sDhFBX<+s$WV}i-2VnW~cwRCFqY~wNgTjMMeteaQofMW6&A&7*OU^)L zQt<9W|LP?Zv13Z^e~j=IprEE=FhVNHHR~+$VqSbpq{n_HdI{TZXq(&f_W}uKU+`7$1$JeZ19nB^$LWKro({W@p)%+7oyF*Pr@Byh0au`AmGtkNUOJ;%% zXw5)>#tmFwzZVQ@fyBS4ZHPCVFx*G=TVRZkAPR1nBau9hfTd2v^)zU@S&tDupB+i5 zftUdSfayEo*m~J0rWi&j%#Q~4$>XS1pkaca9Z!!fS@t={5sag$vbH2ph4W8F533kH zPa9S%*90o!C9YYhBNo1XVZ{#X;A!5khGt^4C~iv9MP(r9UVq z6nXu9a)ym7h5YY)SWY&gDt@e)Xx~CQcquHLl(#!foG!k?poj)z0#*ii*=PE_hs{~G zCR==T!Tb?}-rQeTAcSW)hBImJp)1u2mRL#hY~@^%`7&I-ic z3j$;7t+)O-z6V~v?!JW~Wq_i4%peYk%)8LS^oXi1m(dCzwcy2kgpX z5KaaQx?bn6Y&CNCD-Vo<0a2jpOubQku!2lKu!dY_=Kal2s@qKP*W1` z)kpgxh$RsNfB_<8cKV)Gel7@$V_=EV>#wWFu7gz^>xeRoRS`b)>JYy}Kykateku;# zG6iJ*IqV+6a4RA{6(;WBOT9*BLh`r&k6F&`o>paz` zPXsmVrH&5eA?$eCXT`h$SNR9`WP&r6(0cpA5vDDQ=}(nxFNRJ3>beBuIL>+YgEd%X zhIDi(P&Lm{BTuk25a@Fa#^60DE$wFO&`?0LT1%LEjMto!$GvHfr0X$8k9=iLRz+Fp zFGGg!=htz&c-dn1_%%Og^h~9IbI7#SF}zna_vSe!N=C#N1GSHfL`Qi(?KECZ9t-|C zIFWtKSh8(QR4B!%7&oADY0ZESs}ji{q6KZ1v{Z$K6-otT)Zpl+>1fSG~h4sh`sJc0D6wT6km$icKpG^RQyVV{klSVo+RYDHgYL&^3S;ka%o>Bn$(v zP2TWrCbM9Vi<8=W`Y|qg=0-NimjATlbAJ78sR?HoEEV&V@f7kY6%SQZtCqY9%!7jhFc1D_?Qr2Dd6T(h{+At7wz$> z`!(W5m;N#g53JSlx+Qj7i^C=opVN*XP+yYccCvOW;*qOhIuE176%SJ?POiI^DE+8| zTtg&aG#R2Dk^M3$Nwep0ky`AW$%yt)zI)e*pUWv(Lk=*1+hLjpFBWnPJz{(>p&}Cex@k>H;)umC+E^ z{&EKyrv;22RU)+vme<`u1K{O;sYl?loJNrd$LT89Y1xlP{I}Y$N|YSPQFYBBo>oq+ z#FDp{as4t`0Fn{5k)(ozfO|3!M0pN^{2D z_dePYZ;UD|1uu=`R<5QMK*_nR?iU#IaCQgAL#8ZOs^`fs_U@PMF6KF@Wz%q?h`4^Z zFOO5tG#t!EX8;Lw@hXEOz51c65w-H?;NfGPpkXjdp;Eb5B~N=80r-hjI2btEPfP;7 z?f193T%K%+gRhrNDMow&&QTB`f*?$`gDxgFxJ18X-z0fXJL*7Id;cFi`j| zccS<`0~2l@N^l`|Ec)s^>hAiF%@O~ga}pjYzVT-^~OZ;idljOni5f-?yDtN zHR8jCEJI&q5V`qTktqDF3i0_LN)Z$`H zdy=X=HC&g^hI*(4`E(q%w|T6$tWqtVv>!0S|F(nV^3L40PwQp>CbG3{~fNC6~OXKva@9)*r=DoeA%8e zTh_-{R|5cDcm7p8zI6jcHNEzp_z~(6(qa{4;^Ooq%CvQhwjbkUKDo;lYtQQasT|S> z8DP47P?SsKlF-tgDbp@nk0ud-F*P-fB)xmncz_$R(sp-sa=IzIzgR;wQ?7@eynWc_ zIos|CADziViqB%mfICn!vm}_#94aCCgY^BAo^k6->85zK(sOQ`rJv5B&OBzZmMxcu zh-x9fZ{LuS>EuJCdXOtf1*i7W4(E{prX!L}!6ch1 z4Kl~3Eh*|#JURdUTCj~q#flx4{8aiq|Gf1sPNZKwJS<1fI9t#rzTPG?Qp24;qg5~j*uGs1>}Ta7Rg{HiGTs|>v3M^^ z)FqMyc2%9$vmVD)8gqPv3SG8LN7H2+Dh#j6rPDdF z(|o>zB}Nj($3EUXlYaJTeW!8gbf>AHtqdRSJU_C?gyl>rNr;WBFk8vRz@ zJCRM}dNHh4grM<*yoNiQ!y08=mHzyJ+}!A<=kCK*d)uG}Ihm%b8Qyj7+5<;cR@lW8 z?2T}Op0~QRbcqiK8|1mTd=parl@5zewJ_ENmMXc+e!c&*dWG&hJAU66#IFKpq8-iTUTuK$7jKd(D>q!g6}; zrqz6;p3}TNHq8!Oi_at0qnd&8wCkZTz&=HLLPmFnNv%~cjZZ1pp2kh7@Hp=Um_uxv z1&AKMMU)u60hNB_$wRic=o7+70PPD*R%f$+##MHaa`pZQ8KY#2qVBgPz_hGXKe#;t zxbn@ehadUX%irWXc930BaoH^4SO{wiV0R4(`s1Q7Gc&&$SO1*98I(#(E>@v+q>I6j zGhW7z&%FgNSRRqW)OFkO6nuY7z~)xiPV*Y^)cr#>vFsiiiBKRPs*og5lgjG8=;m#k zW_gjAU0{w$FU)9kDeBXh%%Ib+t=umG)R|3#B&!rm0Q@Et7R>IAn;A>1Zj&M8K^!ra za!*soWwyRGdVP7v25Iu(e4J}bnV!E|p|SHeox(1Wo;W_AQxR}GRhx!w{Q7^zhymbyb-K=u?r8n=O zuKR$$^W$@FmO(;9bTpb>*wL51^w#D38Mso723)~guNRFUUhbqcS-r)iSOCbFk=KV> z0_X0bIxViEx3lHWT5Y1zT=+VC2dJ2~*aZsHW50~wM9KvS`<&X`+O^H2P)&Mmb$=@o zwv0Po+eAhoU?V2;*2ddTeEIXWy@BJr@BCzmP}xcXm3;b<2Cc#}rdSqXIoWqc0SWhwg;~_T6PgEiMbO z_3x9V1!EbA3?=PbSmNZv^P&lxcAJzhS7yrJO!jA+k1Q9Ars?1QpW?nUDyyz*mu>`Z zT2e|t8tHD3Zjcs`Mgi#tMY_911OZ8Dq&uWbknV12_|}cj^SY8 zQE*IBDauV2!sFLwjns{2Hls7S>#R;NUu3OV!1RP(Bj@X#nDL9ar`jw%s#$>Q@PJI$ z|5Y)g#xi-VYaJvGu<~I8dFE#0P@HlEiBpWdjz0nLR_R;^R;e@c^>BcnoPM+>Pa4R z8vD(xYBZNO@o4@>eXM6ad1MZCoMx+UnJdhxyWGwBaOG21qh3TKUjI7TXhoaIi;)O2 zncp>XX5U~n93J9GJVsKchYn( zJ{i|1#2Q8H^m?mbF5wvsQa{ec!PC^WJ~KHQ<3vB(O}DGlf?qZ`A@M(c<3VuL*2jXe zioc)}ECZ>6SBt}~S5Fw`aO+pj`I>`v3{946vntV30FGXQ93siqL+JjxD)ZwLUf4!c z=hjzM-HyE!=xrHo86|mxj|&mcDqNJvA!grb487o_X?rRQy!iU- z8@@bCWKccFgECfM`xwJYY!{(G&cPc`Qh)&V_~g_42B~mv`5Spr}cdzoF6j2ualO`e2|pGFJAxkl5&{ zJW4pj-;}ps@**2fcT0E@duEo;19dXjOwg#F%V3h&d-PkNW4~2tUZ<+gnTAU!4?aY_ z_{%Xq#C=TJ`GF8C1|&x)HIm$};KT|BP_M^^OOk++^(TnZwcdNYY`_BHHji8=TRT=n18 zvSb`*&l1HQXjpcL%n6&vV{}jza3Ub&f?g=)j~nV}$DMl!n;C+`qMvD(()SxV_ROd` ziEYPva#=<@14u+*s|P|yI|(J=?HbiK23~9!hF2g&)O^rdsbxs zt736SboKWFh}Nk<4gHFAvH9*wo&uJT_=s@B1o!We6A-fj8YgTRyg%?b5zH9Hm^`Tv z*60C{DD88ZBlndk-_Hc<#Z{VU!1lE^p&!yLN_EA;;Vd>S_GDS;q60TStd5ElPXsE3qhGS$dDxqNQi(uF*= zl+PwhjTV$Vy6mI<)=+zpGF=S#w>Wh=g+gnkLkcKKgmNQ+zmzTvU9mp-NP1#E;{_=g zP=K<{@d034t99T7G;5~3LNH6R-yokmlDss=qwro1*B<;6Q2c4n==-ltCCn$-+ewRU7Os(j!OVsHk6ofay3s z3tNRC@%vpe`toDxV}k}Cp~H$=v>3S$jW6D8)`KCrn>4`)kl#~F{l{+Zi9kS>6Hg|V4mTB_6dB8hs?pL2a6 zk4!O1(=MLV-*p%5x~IVy{}2=OWCHZ$B_`Li2eQ#74tDLGQKxLHfb;I@|zCVh2`mxm1KUOoTl}J zj%Q_|fgc}zlF2F^E->&kSkLykeUhW5^o|Wtmz7>-pvwO81HTAwex$J1>DMieEFgh| zA?YX6N^HqC{Q%uT32SNebY0rIT&tD1SiJcauB(lL z(xH)`I1d~!@Ox)vHjXrSV%@%+#ClQddf();P5~yw6ll6UpL8qZg4!@PBP^eqywW%- zYY)FL>egtUl(;jVULFRqYK|p4p8C_R;0dKX`!w%K1PZO95?9=?PL;On?r(eud`&;! zG3hZ@*@!szxmw(;52axto)ETt-xzv#XRG^t+v3vn#ld!AWZ5gI36R1n!IId^PCuY- z*_0^_>fd=3PSPCGOfWk;4j#UdK@^sbAdcw6XWWKU%aaXhg-4?DthSxb4MqAlFY9?n`LW_`cDg0WXEy?Rv4^41RcRU`+Sd z1DVh6ad{U%tT^qqoE(+?0AEp5m}@VMTF=ExKfh!!jX~R`XXWz5wc%Zspm)v(gG~Lp zBlC?&+2jHAjlgmBx>vNPTffj(o+<}~Sx}9C)gzHhIQbLX8H!v@yBUVtN0|M7S?Z?| zR%&7?u#0cXElb5{lckAVLDGMjNkJl=xG#Ndx{SQG0!jHCEo}n0b|PdAD&~7$gS7Cnn#zPBX6{w!X&^i&&DDC=x#PUAvibX($ECMS0 z#YT?_Z_#zN@!ic#+veI@0Z*1JnRbbxCW{e^gwkor=)J_vmr!8dLAa3s7G?q$X>ZN&;i}oL7bN|1sRZ=>i3p3{&4g~p>-gapB;rj7 zxDEAB3Ibs#DO6!zF8WH3hm}2rrN?{x-0D5x0IjqPx~B2uDABDu_T|MfxjViZA70uV z`C>Dm*%2#88HpW?3frSd>9D#U5@Jsmwt)mNpg>hCszuqf%UWiD)u`#c&Tcr7nR@Uy z_M6=?X=z*XS)gZ>4R(fO?uS!(5-Q5E2If>Qo_As23k%F2o+nCz;qfPOwVmepd%O<) z9gG9^TK0g15C73ci1u@U?xFuTx4p#;cIfU>Va9Ft_!vde9+t!!J^l>!czwY9=#K#a zYua0Dn;z%(GO7J({D^h9+$r$1CJ%1xixNCIaT(M5=X@S;a%g}k$J38Ojv2zPTX2sn ztzZwr2;n#rne_4GUk^$h)7+E6z*i_wsxh>BqXEB zGVmIUB1bO2`l3=v=*H<3Wze8LxD63)IsRPtTe1)Z%h8HyUwVp8`4a05yE(sfI0bq+ z(|UgbNmHre5((}2-An}z$E|*`zEt?O2ucW~eq}!J`od$JZI{0cE9Cdw$Vz|Y#ZBBW z*`SY3K`4uMHnUD0MmIoatR6XQk!fiCb=TPWV0yc@mZEev^n9*}uOdl9m#Ye?k^qzU zRRM>kPQGwoLEsjx*Gvii;8#u=WP16u;D$^IWGdAh$**d8q%OO&fu{LNRJx|Ht2I5VInz2|emi<6ER5kI?q z=k+g~)8PY5zTZlptjy=ExmlN%KECG94s%fnhsUi$Cq7aW(r4^yAr-#L8N*biE)&}x zrolSbKOj%L1{QqLzp-nE;_)7zyW9xAPOm_koI-9dohdWaMPOb0#vk8gvIp3?Yn|%O z;qk#D<-a2~1zL5zIsBaIN#GJl<+;$%gIL@V-_?uh-fwI# z2TA?=ooF>mZl^zU6jEWigq}G?>{wza0;` zpfrLdt8uy)PH*;-YaYxJU{#}&C)d$Sc}0sYVkmfP=(3c^sD@#~eILO@%xT4bVYTN0 z?Ibu6T{<|3!mt`c?5eRk-rszd(w0!>(5=Ocu;z@)WtYw`^}^kX`*dfN`YoE#UiLW6{=p2(`@4jF~}JF6RfgdqUs5bE*ro)Qf!LF%@24SO6NETIGv?q;cRg z?I4-oX1#xz#WPvjjX_$tFjZ`b3wQui;tp*CMyKPIP1s#ogseu(x#z?gZ6h&V2W%=w zkiR9<3}}~5U?c6%C4({de-XlWhf9_m(?)Z+qY0{{37Dn2Ol#jpE0SH@nU4`9-(Bmw z6*0_pTTApDjpdiS%m=6x-K~D+zp(Bw7Q&n@GU&$MGE3Fcuj;99cchpar8?*VF%S2Z zjJd-=X?}S3oggv9b>)wu{trjuG{RyG)kuaM@xYn9q$@s|#BY}RrDC9d@gVE-z;1wg zHiv?C>4TzwG4Gt6d;PIoMU^clDwECi6T*^+Rm743zs2PzOQ6&dAd4YUijVh+j>@4e zJZoa65$n{WDLW(fCk4mEGxdf7^}Nv}Ger{@91H8>Pwy24YQ!ed_<3Q@tjEOxkH-0j z0#^THv0sSk3YSl8uN)*+dt=m%(63~?U^0bWx!D+-W}KOp587X7p1o{oL1({hgQj^0 z+%+BVG?_$b{AJKx+bl+u+_)H#hq~6p>5MU;V@#9|IJfAhNuLmBzAWqC2w(v9H};O3iV zf0)B7n&Zk*4nh{`qIm^QcX>fXQVgkuLf7NN=svS7xj4QPFQn;R++TlP6#UB`QsxPK zkBdmhE+x3L5$DkT3W3jQ{rPT8P*YP=Ehtb{G?-WNobF#Yxbu~1vPk~Lx^9D zw{CqQXRD&H;L2!nSt*UNJjhw>tSYA;M9m%?AfyZi2o0f&n*CGuf9UD||J%waVMRnl z6o7W0W}${ydk|*1)fgRm;$RNQb;6-aR6U>Ds~^as7}(G<()vKk=cqUm-nW!}iOgX@ zjtDB__EqH*9Ker~!&gQ=qrdrt-hR6`ZV>YZ)$f4Jd* zcL8C^?{EL?>Ouk3ptdSiWj#^mdN5C9ix~!`|0;s%D3{A}hStsf5>S|onn-~pl0?2z z#&E72F7pjYMc+$?KoF}bfs|e{$Y(+M_d`5`dX9db;|!zQ&Bam6b137+m;#zXnv(uP zhejAE34RLa+?K;Ipw5IoeeM+CDO9xzfm0IUs!`+)Qi%X__+$znZFu@cDO28c~;bgBpRIW~AKtK3W( znlrXA`6sa&E=fCRLaR7NJf6(#P*;&0ekQfdk z@Zj!8PzksVr}cc02Qz`ttLI?=S)c$+67EfgIS{yRa{2Sv|84=u)&3V-s3cxZ# zP)s<$$x?zt8I#1`Ffa?%axK5O(|&3}M^bbZ<%ggy>U;gJd^1O2g5--+)%Yrx}cB zp}*kn6W+-lix~T!=cR%G!e5&24Oj!#F^;+@Uv^dzYEU{%ZZsC7u-Y;%Y zM5u))oB>HhW+Huk{l)EV>sE*^9=Lx5dcXJkX<_L7IB(y%ey|7}9bs&@OgYeRG^?a5 z7lL+08W6i|ZCj@G|5xzH0tjlG5Hla>B{p7IP_hG(veI=4u~WUh3=i;m*%mio3=uqZ zwm8t8MUXeXmrG(+kBZjSOnZ6?{~q>0fD-3 zQBcIFG*?!-luhF<+;dJMTM9}8N=2Qm+y82)E`M^Z* zbJ!9>-`v3lHBOV(07N&SSJ^@Q91M@>?A{hcN@cU4RnuaLLD{ zjQ_={i3aUUaF{&!j=!xH7>dq3(0BFWfMCPFjChc-W`;n(5+tFj4kyx|D4?>6rQ>-uHcbai@gDSdwfLc5 z+|raEEFR?WK*w`l_^G$3(sq0Or_VITpFt1p-Yri5t1<`8Y#70}eT_8*R{bwKs7w_) zx%_W!x&Iv$FvSS~N~6*HUxjQK)MKQfbiLb<|N4~yqJPwSg?7PV7k`CN6(|$V*dd{C@4)v~Fx^H9( zZ3(}R;-^sij0C_W5mh@=OKqKDhkyP^k9DWE@c7;}lw9t#A(HaC!O+SlN{U{hxn{U_ z!q2+ycNFU#t>=);{Z?tT*0i$_?qS8t7Oh4kf_a@5NvP%rhgZggeZIr~Zu2*4`A4i2 z+!ruB2Vtd)3C{>qD{bwCeKwMi&bC)}_>8jMXD4|fuH#;Xexz=A zCWGP1XZf#Xg9i_aP2OzYnv`O7O@|!pEGolgk`S_(9{rHjzq@{e@%Y#y=zP#Vpm?^r z@MZ7Zug2djQ^k7aXe3(2UEvJsPx5euZ(3N+P4e*Eh7pKhge~8s56Q!#@k$zZ7HZb7 zm)hhH5(%5pGa_q=mu4!j#9=-CEYaXn@6MCs$-BC8a8TMmUg$30xR9gkq~GA8NO67L zhu^rXcU^7g{Bx;A8`(_f4$#)Amg@{H^Q^Jwj@P9N{2Zq%ox_^!mzL?{3-Gd|Hkj40 zS(}a_5L{eU+FYa({l?HNQ`J;s;6BPlScP@VU+fon0BKvD}BQl^c(UhYgLX zOt=Wol;LFVjv@k>HNAc?uCX4%iX57mnjfq&-KXSmIA9T42rhdeBfa-yxmJb!9)c5O z-}8qgtFj9VBJ=0ArW&x0JvKTQm)cdP7nFGjv#w5*D{bcJ>_!{@DBm0*uS>Crj&((u zv?V+6I{jgi4u4!QcD8dLx-GLAMt%KgagJvp2*EupiKX>4DxXSuU+}BW6VarDixAw+ zqEW32Q@a$s3N3-cwtiuM4?@UubJ(A znfel#i-zhN8VbY+gzLK$AIXk6g+KmUan@~jK}$!Q4FrMnl%9!X`mXD~clAWmBrgn* z8VPeyn==hVL!iAN5wXU|w45^Z?Tcr@Y=+RXiEuz1z4wMw`4uhSqU0noOPP&RjjQ;- z_R#wrglY1eNVQIlcHr_D{ZxZ1t=V?AB(U{#^n?hkB|g~x`v5Z*th^ksvPAu#!-+A{ z$J&LVW$ynDYP`*}f1HrrJ<9EI!z)=}tb7Ik!tPgZ?E>~=((qzN?0CA#@amhw#G%Ty z#aS9W!DW6!-if?n;5?V#Vo!swu+vR-g4R9#+iS~WDSnM`47($3)d@&zbxeB$X z%=N2YS1HjBG2bF*?TzUnmN5hxbV#SRE|794#$;{gb+;yYQ{k}OVF}RTOHjA@* zSF7p=H;HOiYS&ipd2EfQ;sF!jpWz>{;%coFfv5sri z)IEF) z(YOdB!+oJn5>{z-jOg;OFxl|?Zs|I9l6>s9P9HsC-{jMyp)a}llknIZZ`JfWJZE_% zTl(Y6%DIDP%%M3CVYLFhlSBXOVi*qyg5GsTbsGadh6u z8!X9KRsmcS9N>tTRIXR9UL-U8B7x3mYxe1y)mUMI`|;-Yjlr#u{J7Xy%0SKBNxQ&p zU;5U!M5jv)Y+Ww;o>xu|GlI3s{?}41$=&GN zz5(3cq)2|;Yn8&BNPd$f{05IOU^4a-%0%&i1JjplA%~^BhN&Mlg6V$&3!QW5Wn0P? zm|>*gfq;j<=)sOt8w&$3s_mi2^b9P`zxS2-!^ptS!o=u8A1eFkhmjl&OZLgS*qFG~ z6-ZAt*i*x^GBHP0dv7Q;zJFOftg<`PNODxOeDrqC!*$K{0Dh-=A$x=BIi^DS`E9ZY z*y42kV6)9BGQU6_t0|n3&q-A#$C~i~IhXH(O7c3wkME+Y)`i5ndKZRzHJM8uTQ$jh z0wbmO{T@nOwoe8%ysev zY`ogI(Wvp}U+$d8}c%-`h?j~V`C)O z+^KOngl2d0JF=NkVr9R244J5Ue*44Rhb?KPHvW;dVF zz#!*E!Q7tE?jOV)yX@GX4~u@xb=Vr;JMC&L0)SB8(jv@bH!m*Syz&#>u5T|Ko_}3{ zS$V41aZRDnmg}&h+SUC03eyeWpsm5AJFgf@Uvas)!S%Vm zX0R-toK8^Ic=vprHN27tW-uyT^3rjQ&7;GUzjVjC6H-v6S8n}fTYcIP*>7k3acj5F zEq;~Fj@L0g_^}kx)K+MBh>uLs6^VVM*b~+ zp&mv^MLoueW81_;FxHT}Y|%(muz|CC%qN2t(9(zqiGz#@@!1_~yhlP#8huO!m=WD^Xrdhp$mxjtpx5sGZG(;~Q}P(8h@)q`9}i+?bm& zzJ2DsB$Y7>G=@%5RwB}Pwr)X;_{qWziztGjG3Si^pkKcD6YmHvSzI9&Pp13-f!!O&@+}RBZdY>Abd#zU@fK6hv z|GmcX%#}Z5d{-`evP4nOcBX+zdMhc-0m6?*o6mp#{6Rs?p-_o&*^jNExI|s?2p*`HQZF0 zJBZE8w2MniG5d{J1^$$yZ`hl$uJ6B~cLd_S&Gr*J>y~v{ymc$WK!6MxO1}EBHsVQa z);cywx48c39BzJ4i!req5y6bc-tgeZh0obbulsiC8;4^WGrYr$pN)ClI{B$KHFgV; zfC`-Br1KQOW+k&iUqUdI@uREk=A_RFf4JQ8*MDWuH4|{X6C{w5St@O2SzPisJ9LZ7 zyQ;mZ^}ep&m!ZF`w45ZJnsAQQmJ<@H_i1J`bI+8ZS>Dt?yiZ0K=*S-M9;{1Xtec%C^F=q3cA-_Q4F}^gU`q zh8IN+To9>AJG&%3-MdGN+!sJe+3!rKDP8vow75R)+9wlvve)xk+lM7F11=J~Oa`V$ zj55;A=?5}2qOwKl@$$78f5*r{m{o-AH_CJaautlbiX+z+d3s|k*h!cfiHT4_U3g2x z%AqzqX^u0=g0YbrgPkAQPDxjOBw`12i3i-e?T72`LTv zko)i_^uB6bV;D;T`E|+vxEFnKumi~dpd*O=_c63-C2(nvDFydG?&U2iERr9~QL2vY z-hESs|GW&?BQ5AbeTU7f|Hn?Ht;XBN=_jN9`_}UazgFWD850;wC$$}p z=R|h*vBl4DO7s5t7|L#hT|5G?GnDgqM3InOyx4!z-u|~U{>aKLmRYqe!ga!KrQNTU zZQrguD<`Joe28U%|E%>Jnfu8C?rCTFre#(*bBVdXuVvQq5Y9s#Xe|Z8yBanmzw`E@ zf!~}&f*O@?^11H^68sAO^|w7%uz~p>=R#W^i#CkB<}@w1Rd3B2>BmJ92*2umUEr32 z4=3#^eWG=Eqy`-^tX#iViB5QzU4kx{;a_*(1n=$c5=L4K$rryGTX>4;vPL1C{Qjry z6nwU?I-BJ3CGux{}0=V$wkp|HWc zOVw@-fR5K+bL)m)3gvm%pb@Yvw;{NNkC$2<~&PBbi()t%I3q`8c2Z5R!VZ4FHr-E8c@H$d>XaRD!FOq>je-E6Es zIdZx2lK#1a3wRA~W+Em2bBmJ|FR8kWJh70ig9$MkBQqm2=|^~CVqzW#V^c0gVbQ<8 z4!q+fHFt8d<6>fRb#-NQea~p?V8-;0larH)nT3gkg#oyO!O{JblYtwAZZJnKXNlC#E`uFFraXML;{`aI$j(<-Jm>?7Q6Q*~J%uN6G z4Sbac+{z{AU||A08QlNJcRYXY{6B4fKZl12{P6!7%wIG8(+W)WBRmh&zsvR!{;lph zECd8UgoLnwvK!=KCbXNf7~IsrLbB$T)xl=WvnCH!7{)y}u}8@=Wc8_a0p zk>}5AeIUJilGD$t4_=aso;Pgr?V#dY@7RnZ6FW&9Y}qhvn@R&~)ISU1H;Nj%xe>`> zpR~?T6ZDQm8Nw*Q0jd)9_Y0Q4T-L$zN$=9`3(P%lz;^2B_3Nf4D#bZ|O^}VP@d7Ct zF(g`*49I@QHdqykVVS1_vW|I8T%{6&pCRwDz3?2H#)U17^R&J*sv&Ze6!OpQBA}B* zMq(x@1WbM@^l0g)nrpc|pS%ynC3k~~#=3YED7VR*=J`TBrg6a1_>H2u-H7hJs<~bb z5QLrxu*+AxYGtokVWOqb2nNp{d=WPQOEuC(PooUMAi-Yad>})V%>E$Spz%ZQpAAFl zfimX&VhyT~m4~cDS)f+2GH|p_T^J<oYnc<^%3kd+i#(SiT1RzapcL2*O%M=ZBuaeTCc4YSe#agoACy@{f%99idjm zs-e*OHL%b8z2JYgTs;v2lRsj^Wc%L{kn4K_BC@pdrAp*~+7tj$V+c=4O88e$fF5%| z=AK61vA`?z*X1Vy2yGp;uBz-mGv)_1hyh}I6a}Xx{9hXu1u)J$Hx15zz6b%gKnMvQ z9OeHHgIhD4T>5)#ej0E*zIUKTBHpf6*OMQbm7KJW4i1881vG0r`LHk~mv7vm zR(HqUd^P;NrS;+foSH6f1%5!TC;gD^!b;-XD_GVuyem96%OWWPnG;5reFlrAVWc0!3&e1Y zlfJ%wWcAu-peYYfB>FQ=)foS?`V7G8yKp;MbNtsP90h^dRcvoW^S1}nZzBlV1=X;6l4usu2{?*## z^Tb7~@-0Y??+HUB0;?|=5zo-zWRU`xm=Z8j8H>{|e|_Jd-F9UyhVPlb!FrL|dV$&N z=9g=e*BuL&^L~)q*#?QpaN-iF+x9Rf29qv6BJKzN62-4oHa{EG#RGN1;Xols1cFCP z^>KW@KGNAA2^^3Wts{UV_Y)Vn;g9C{A<3Tzm{q3BlT{)DZkOKKQnkr;Pgn6DD{JwT zGPu$i9H`ig+Wc8O?(+Gfh%=RXM24INvHZfXLpRiFO=XlzRb{jiVBg^Q!J$(Ui$&ur zLy>`)VJ_q$BLx=P{_Jq?2Y(WwVDrzibGIs$x&##!6-3s8_m_LlD^7b;d(Qp{SoF90 zZ%ys2L51Pj>T%lwLdWfeGU*$)XPY?OF7%iJc&#G9b}sl;ae}u~UlwEtc{q{#DxO>- zAOIR6Pa=Vm!~5RpONx4{3j;hRZBR0UR-QpmFtt))D z$X;VNrv%UiIPQ)i#)l9YCIh?mWqY*r&-O;552n7S0y+pL6N~28?%WzlK^{$KAGo{N zjj~y6bASKwZVUaxr{#s>%c!^SVY+<>qoCdRLE&AizAsV0^8NM{E%PR7(fg)a{*4{) zSBtaMYC@1XFR-EI16gX6fF||Fp&-WMuRcBc*@zrN$UAlL*gHj~+5zmQIYO>A(px?s z9>uScnE5uFeTNBtO{gs{N4Pb{LwWO$LxFwGj=O?RJHTLW&jyI<4@efZ$t#oS)Pe^G zg1(Tgp*f~(B!Oj{*c1-x<4w=>sF*)HOa3AS_#A`zvh(QHGr$zshd^eo^mR>UnMx>R6TQK!S(u63#>acs|1P`)4DQXcX>e zQjzeSPrq80M&+04EY5z~c3LzF)gY-Pc1(Sh3ekBx-irgA;CI1k*udclwE&H?5e1Gv zX1?>*h|nD`NlRJW#~#tUK3uFv=69s=o#H5xx>>K?Qx0|U$dmpJpQHHfh2EZIkMB^7gelk zE{Y!sD;|jNHj_>wpeOn%n5`tqH6Za)5Ke--3`LZlRTa8gKN}vg}%2f#uA$? z#g3{bw+X8t1WBvW#6OSeX}o-|f>@Glmd7KKGz>$dsPejVFdj~X1N5o@AQ%DE0qmca zC*X515S3Mx6drrAP5AjCL?dxfC8MOnX#j~b3ja_k1e^y}P~M~a2L!F|D$SleyLC_Y z3ejBUA6V`wte8Bsrs1a|A_B-ZmbXHf>gCI%wu5!fZI5r)35Ome3qSwji+F6uncn`` zyJGjzYOW0BAnWk->G|QLo^+wv(JY^odb%g?mO-}-rzZq?K76&y|JB)|S++n~AjSdM z`tn}4{bS)5SRupc7$%L$H^j`ta}N1aoc#^Y@IIK!qx79@kEEcUOIghOSvQ?6LtFO* z_7wg(ZBB5Z(55fXPaeNij1N4{x8P(mx%vpY?@UIs)gfOVFXRZ0M{`~e!{pbR zjT>J6p5%fb1vKepFj@6I0gs#f15)Cs`2QV@dIo=|&r_BCrYMe!zUWrK2Aw=9yZE6# z5INMDPm+@M;Zw%w)-EX9n8O*#5hH@9ac4`20v z1?%J{V2`g;V=Qs_t93q3eo`4A8P z`AWS5I~}mbuk3sd&lL(06!en7KppmlGCs;Jqlp@B>3gvZ;}yHwpY55<7rWwp4`^}X z@SICU3_ds^^-@)aL|PR|vE(>szjHf*YOrfEXEaU+c7FIHQN-g&KR5iAQ0G{zr!i>W z@;;a=zrTNeeqJI7k=}kf?Ll73PH(!7MRBOdj9N>)W}?VdONgK=b6T=Zu{4UW6A1mC5mOkmBv2ayoq zXtkeN`~5;e5s}v|RV|;IGqZn|w#q(2Bm&F{9(qG4*j~{~f*pgU4XQXi?7hr!*7BuR&JDw|vA-6IU^GqTfj+Z&%IKn2#>VFzWEK&k9_9R)md0FT)=%a2( z@u|v;mH~!k6ysaxVOu(!?IelSFc8*5vu2LSjF7+&^_oY|7iwJXaNyuQ`z#S53ZEQT zlUF>XIMJEB9ct5?TEIK{k_u>v_peNx!D6w70tH%KU$6S?R=uYld)%QqSM9cc^?M(k z{9yXE-A14~3jZTNh)uAIZTG|)3MpOSk7tU)0k*4uuFgUC@Fp_<^xchoN=!8f4T zfNENKnVBcPF@DkhC>aiD)l-sf+4H7?>qwVun{v?EOOZ}j@y!BdgPrf^KILOi57*Zb zvRl(75TaRcrTz?e_O)9R#oU=n-)tN!g>GD=-J&baG9@|v?q?3T&ClrrgmU2*y9@ge zORC3}6UPV&&jsEX7bw7l0SA&I4eHZMTbS`seD~}uYZ_}c2VCzb5DjmFfbLwfA0dtX z%lV0$=Q#UJVN=1mpV)(CZkAEr5n5H!!-xx9Uskd`qWznw}Ea zRV?HR2Ir&eqo2j{xgg8AvamMyi;a!Gh})ya)mTcIG?dXYz0Qx%ce_MCzPx({KEUB? z1riB@TSg$T&#fNl2R8XSTZryOv3~7nzz$TBnVB6O&y=9M9506f9UDE~o{0l%*Tft9 zOIIeHt#7&6aeZeecr2596VMs!$?FneT@nfh@}Zrp9yk07@y!(;Zh;OGh==FnYHM90 zL@G6EP1UOnp_3VP!k?cW@^xBWLW%f%ig`Jm_Y-xI0l(IPSBGAyXCQ`1Z2kLZq0w<+ zv?>u$$gI#v&=9))&f!~fi4A|vS*skY0ohaUb0}|^YvlT&p&8TScYKlGI|5;ueKu@4 zzRnH#TuzD=t2V=T9L>D7-|R;O;mCV!=Jen@wA}mq&ydf2#8I+9HrdMbE+2QNRmjbU_yS{xD z^;#bSm>WF5NXzE&klxi*qRbZG2tPlFNIYH`LWy{<;@9J1{(yT)&2JPph>ilAl?O4F z#fzs@DBTUL*E`c6IB#&6A44G4AMU zq2Sa*y7ud>ChAlWj+z?8y9Jx<&Ok2iC;f8-Ki{3D=)Ln?S(e93*3}rmo3#xMpDiUV zDWdwORvzp-(-T7-NhO#4{m^1MFwyh&cU;BDO_*&RMpxR^!Ny)~QIo@tSL}WHox$L# z0Y_z%jk4t0E`#RxzR!Fb@7@u5A|E1OJkwuEgo)Gn4Me}>9~>+kr@yB0vtlm`q0S}E z#$X6`(QBL>cpPtox}7)H%_=*31=YJTU#`V&^OVkEPYQ#C9}3vG&DN8S9>AV5`#yUq zS7^U|xH%E4HthQ-k0+PlXM+YrGSEv-kD`1FZnnStL3if@Ww_3`?7chXT6TBmCQv-S zU-4FtT6J%2hRNx*s$O3m%*|S~-(Pj7v)i5y^bQs4)Fyvq)GKE}!sof|rY1IY#t@O9 z4@+;bo_l6JDCT;T+(XdnvT=ClC|-uYuS%^(Xn4k5K!3=j+u@g;ovo@EQA(t9r}e8a zQAMF^8LfFeVWTgOOv|Os>kcg><}r#swRUZ|zBD$3t&&HVI(H0jYbt4Eq1g%>IAI~f zm~`u83H2Smr6DawFZe>|TSN6C%I86phLA~CzGY?EjWf80ds5}NtU@EhAKs&X`MB*m zcrY|gC4R49xiyr8M}_p#2(>$tI?ds6_=Rv4t&dLtX1iZX_&EEB+m)YClDu5HWGYv* zjML{F%FEk3j>-0;n@7;{yN`~?_~hcT5N8|xJ;2mTAky5u5N8!2Kg6ziMxvpFj05Ie z6o>$s4ZcHN?$1hT-5Y1>?@fI*s$*!wf%}r~_xSpwV(Wb8{Ubv8qr_9*?;rP@ayD=n z+cZ2wdDBu#;-kC2Vq*TD{fWL?U-J8*NJ&+<^#ret4sHcbs_I<$d9^f0nnW z(PKdv3f)*acjIKWmfGP;4S_c-VpnJ5r@hDWPK%Jt^oBgGlPvYpW!5n0o&HYtdb9@n zS=<8IOk7MWWufDYW;-EHJKyaa$rP=6w-VKfou_&TJy&z&o8GWl3|-=tHt$!QYSmuQ zZ*F~bP#0NiAS;M3H`YAt?4#YIu zBy^2xz3c}=zf4NzCl)!F3|p3JxiQ#Ym{`@>PGhB1>iVm{Lhw<1lj>kHFoikNLublE z#rGYy{bp@!_CP#^jbW}#t90CEx$W(cN^d6Tw3*?HtgY94Icyd!-S{>NRjhf|WXd`ARgumkS zjxIah&)^xexB?^h-s|5VYH)m;gN!(5cUsfe=JkZk!x3L9C{c=X#zpEKZZckE!BDaC z=?hR-)JUDPSx;DM}`jJT@wOsU{#@?=V8(hg{}T{TQlT!^OQt1zs#S!$|Q z-8U%_+*d9Mh+O!;D=9v#w{-AmFRXa{hjL1^M>t^f{883 z<()y&l|JIadn3sdbZC^)p@+i`=h`%;qy)2!kAS`-1&#)_Qc1wG{n$YgyRb9M_QPjQ zN7tQ0&UUP7i;*3!7>mdCDGI4Qi)|2Xd`u{{Z_!xBD?VH>2k>r`JCj7(w`fxx8ZeStbI^*X%q{)4(2PX?S@FFt$CsiKn{A( zT&86WrtFq0Z=yb0ebYk0xkb8bYa>?ews>DiCejkaET?h2|LuYQ7&7qjblSXpdA3+n zr80`OAiX1Edwy~8?G4fLysDYT7_nq-8ClVn6u`H7GM-q3y=Z&~HBD*pIgNt9>IW^7XO|+D%TVJ;7cveHU0?vgQVYSwC;o z3=s$GJh2>E$Cmg$Ln_>BC$Z2@eHEhOEvHjzHFU>kGp`~kJ<}=l6j3eL$agPSjtdaB z?n6uLqT}UBPxf!)jvsWgErH*zHu@sG|xB7POExkQzb((_DrkYaVlV-6b(h8omFVq0nZ{JfV=}6vs8#9+{Srld z=Qw5^nnyWYKp)A3CZZ;uSDf@s9qB5aO*CuSotUqml%Le1dY;CY)r%7X-wR~I@OR!p!WD*IUxH0_HQarI7jN1n#_#6*c@;KJ)aXlV2JRixN zXw7`r(^G?A-px$Lss6QEg5~(v`*S+B=JA(imS)WY#uhybL}EhPNX{p}Mq;hKQV@wq zBX-8-rBIDIzfVCUI4@btzOq8vd5nS2;k2l!V1^si?6lgC;H4$y9y$K@xqAa8O%QY~ z$nSL*g~OuE5Z@Vsn(q@tZ_1?CfX`N`QHkA%cTZBG+smF-dfd@D^{29{}R_-k%i8{n?VrO2g{ew9tOP;!4OOwagss=ZNph!Vs8id3n~ zgidY7>LB=%sJM1m6t##$*8FG5s~WTMiuskNZ;vR2qiGU1-D{mPY_)^cA?H@qq*$VN z8Kmy;yv~gdLyAutEINNv$s<3B=}_KCAAZ@y6&C%-u}*A$+<8o9e-!O zaC5)R)G5QCUa5J&T3J|bvvR037@X6Luw#_zaASy~anr1R7n(${iFhzOZ^W>vQ|UsG z+2Am0*-p0@n|(4#g(7D0`y@{f(TW+?-_J7!b?xTaVj2HRqb7QUiHIy^sT*4%mB+ni z@+rdl{xuUU7bM&~1uC9#i!43c9NOX{^>lvTp@^vQuLN}+<9V-P3>s3U56f*e zS@LCJL=IJSpZI$*L&_*3%XqHO6s(|h>RMCROt$Z91pExMp?Vo&=pkPpWhKH!4#arJ zekdtgf+MUxv3e*#aV&Rnh^j2#+36r<@Y|srz89n*=JuMe(1{4Nj1n{5-pB)-)27O~ zgdmD|wG6&S%UMb8pk#4kRF&uN-xBu7^%+1l59~F=*>$t}ItWK0Y+T-`+Xq6!*)vSs z{`q;25*mAzx43U)iXT6V_W4X1H@d0c6i|B`Hs_z4J#0!qPh1aB&(&3ZTe@>rFMq26 ztWtyRY6WI@hxfxqoCB_9?5L#&;IdNxRBuKD0Nyth+RXvkKKBy5GE07};}hZ?ewf=3{a3rT%&rD5N^7+(Alk!mH>Eca1=aMI0Flq_L1ibiEFGy!UrkAfNOTwZNq_H&g#XJ|KE#_&r@Ac~kDl?P zaNPT(O76OOtnN2^(?Wud5>2g$bZSNi*EU?kVZ!$dRTgBuZ|$yVRY;=!DDuSe?-8X- z*TJYt6w2#+>`!D-hVTDM2L+&UlyAUNDILdt+|rW~92jb8Pl`zXZKs(Z9Mc;hf>77>G20^Si`7lkH-Hg10ZMeP{x`nY$`bP9#ifbt5g-5Zz^3f zx%1*`mcz=_rW<4}6Z-d^trVJ75!<}3(9|mRrmY@m6X_~fT~E&D-1|W4r5-Zk`132T z?hR5TQ)mnVXOxhZOnL@9SnKJX!%I`bS-)Ap9tZQ(yk8`4R4G%Bv(~7=jW?Y24B&a| zWNq;`_5nSiD;9kOq-*S`LEAHh5uI2}Q=c@9$%1>eFyo@IV8XE337@}kyzPAaAo}n+ z&u@(C+0@J^jim_0;A#~&V@b3NP@B4DyT6TkTa_Q$y*E-7; zP~-8ksudDCS+<()U`wa&(|wND>~O3MIo}(CXxb!NqwW3K7KREqblfz5-x;SG@gcO6 zqJ$_aAlXE&{~l_in5xs2Z5kKWR=lfy;Pt7e5jI9e+qr-XD<+0lQ4Yh0kIc*{;Jn>V zP}ZDnbS;8554uJt0n!o(ZhpIV0WV8t#5^L3*C;j$cdy1o^yTDUUw_pHbw9kxS%N^4 zqpxb}t70{%rTAfvISKz!a6O52t6aW3d`adJKjh|+o%}2lP)OlW8QPmdyd@&lP78hJ zkZ;lv$Ag5resOlF)>xn23DDz8Y&@Gqy)MGz5dpaf@w-4YK0iwR$l%6%>NqGbGUl7P zok--PL04_bc$i?^JRO$2bHpAq8h`e_wYt{669=?v6*%p0pm_Jf7wZy0 zerkt7uMBe@h;$A1mS)#Yb~(X-AaL2?02@hQTAV+p09Q&`b8;f2*t z@NXxD30k(25GRpyQkz_!;u{qN9);WjVKmd{*9=^ZLfeH`abJxMyrzp_H_W7?5Y!kC zF8HKziuj1QozYBi=afyf_MNh+-J__FfrGVrcy&a3OnSw>j6j(BejcC6gI_i|3%@{<^$wHog?@2qdUljUZ9A}EvHR!7I_!a-iHy2dz zU9DIxOAh+)-}1&9R$$i$;M<%XU=Qg*n7I&i*^-In8;mYB`CL$nK5}8zYqIrR`Pv%3 z#P!@#t5_Z`HNqB+Wr_@M(NsBXav4=leXEkUGG*$5_zE$c){SBLzCVot?`r0FGJ$HM zNma*4mopYcO|e5c&d_|em`3v3P1HroT1>W2_u2a*d(pnPGD)Nq4TnqqrR)`CB_Sch zogc*s&IRV^hn>%N$MP`;So$P?x9m>A;{;%$|7CpfV<6c5Iaa=iH4W4R%NE*9Q$yP> zxIRr(^kKU_Z$Q(2sE7z0h+aGFLug?%Gcz;F)MK^!#FHGDp;2o=mw_+UAu1X6X!MLP zt;eB9XDnkmpP(e_j(4gmw?`@as7FnsKx+bjvvc&5D|_#X%WFmRh)`Ca-j+kX%>(*T zw=-Cp$&5*7Cbq1|KjiXv6o>r=&NNuj{<)&Tw72{yH}ZSB@VQ*y=#BU^w$V&pSWC5v zU@LapwY>|oP$DM6a4hN|42gvG`WLIq$HH3G%iw6ds1KBkhSEu_%}xnih|JE%^pTn8 z=~<%J!>M%Q5ynOKxovk^MBv zVq0U$T)+|rlt~g}Ui=5Ab?kKPS}#sVz2xqxo6nBCtfM2Wv@}OVx(5;^(rUgT zDw4=&H-pN4X5GqU_WH2zbw=Zm%urG8S##jFQ;8l&Ud`Q){Ggyg;rR`bO4q}3zOV1C z7FGo_#_S2zc&5KJ(?|=SQbfTm2m{(~Y)V8KA`oR0_AezMMus9*-t4h!p4l+_472PR z3DL>UHqT?9OBz1?6oOn2UEx$M7t1y(*6%Vb`|}suHVf9zFo| z8`MgZXsJLjOl!fGB*EvXz@`De&IXnuE;_>Ng{%??Y7;?&I$dm>%hvzHe^QSIAfAzG zlTkqC!ui))Z^nAzdfvUZYa=kqD4&&D(c2*-+Lj@fW5@5l$fQc#WV>Lcq_hlAXXK3c zReEK=A=-TKZceV#_Sln>H8G(F`$-hc)cIDD1YC8d}n*Rof2{t5iF!2-Y`hJzn0%GJhS>zkX9f z<$Zv*R?G_dyAkjudtb7thRyl>kY?>&*i?ZuDo$x|Lgt@v4Xx?}B#^&`ok%YHanB&* zr__~9`Qsi;1pAvD)(>_esv!*=|84-9yQTnMnPwXE0o8v-9}nz0_rnB@{?+Rrb0>vf zRU$(y`w4D6eg<)j@DuI#LGHHZlL(0H-JI2rQmZg!*-+19r)%A>np#`Qd>(IKD9(ZG zd6#(C$))pcW=pOsq~rTgR0QP1NWWRkF1NQz-Q`}oO-aLuaMM`YK+ZN$ZJ=_aucOwj z;fQ0UJr{jW6^-NHPNsj~g=*J_m&T&ci7CYL4w{hjtd(%bL0O>;9W9LWeWRV;ZkJs%4<3%90UBcO0040f=#J=w_PH{&&ow@K8<1p%4nZdB|E5w}Wj0Ru!F+;W%aaKR)&TC>+HSV*+$OL6hVbpf zgH|zX8CwH-Y)I(Y=0H9e4e_|Lv_7fOxn}CeGy448By}*lp;Jj*h`}b|!##B_zVkW2ef2oe!^{H`N?hW>YU;?o+e4IkY6?b~+-t ze9VjP8uIj6%=c6ONGH=iB1xe&;rLJgq66e~$}@n9NE`sw;(QPcFhuv6w)kEim0kuz z)RaLzzSoOpnveiZuv!%jiI4|J7d5Mv1O%|>g4%@l_&hF)eUK~NYwpD8Kj+=HwzTBA zU+kP-Un5n%9*^_WWh2JoQ+PCn8!FSNt#Uol1v1k@Omy}xE`_A#MmG|XG=4SRWG0ot zB#E6}yOpPQ8=Bfy?^HYaemP-O%C@auuD3cNAHT_nz_X!T>w9*@%{-ZEKUIJy27+p0 zOQ>jk&b*>0V|e+(S5$IY_{YktU!_t`?{>2fuKeB4w1aKp;NWa!?1iv560RRlKsUWNuc(3l z#F7=B-j6=Cq!MCHoePF@BKdRfwXBxvO0xFRrN(boePaLt5i0YyU43pGd0wiwy1hTB zjH6RyTyAkj07z>60ADng#YFOI7x#MAs#mjH`<)1Xp`TWrdG0_fK)TaRWm2!oX#k}+ z2qf|bLIVUIIFFFi&j{GWlAU|r2AaVGdAD*5EKQXdlsK_DrI>#_D=sU|Bc znRqp>KaK)KpPXWzoCDOJ@mj4~(=Riam1Q4%a^5y%CgjpReRTUul{Yno6YZh~3Ce^3 zh$_K0c9^v!mVZ>qF0W1{*`Dj?nOMmJZR&lT(8q-iFFXn*Rt(=Ql;~)k(GCO zoi6mrmvseyR=6eF58KDi;|Z&Bwr5l6ijc0rtKZyN=#gufV?-2L15drv$D3RtUYHMJ zkP^k6IVVVtKN7c_fFCbIC`3guv472kys|tgB0-SmNir^?xL4uRqOoW$fG4Wg7{dX3 zSy(<$b0$td&dA89$(fiM3=ku7t>!BKZtc z%v^2`#BBCOV0#1;e-M#RP6=(zT?=3R)#_S5?d}eds~+OlhuL+Ts`#B#LVo-lZBj_E z9U7Zz{#Jac1z!8YtpBH&+?m^$%WY;eDi2;SVq}N|;gk6iB>^SlRqT#i%MP(u$_YYT z3LK#B=cv{)7*hdn;&g)*T)H7JRww?2DM4jP|2U%KgX`zAaeAoBx+=xl zt?g{Fu45d7Xlvu7T|Z5fR7AZIi8T#kW56dD!XdIQUv3v)(W@zx!{?Pt`yf$C$EpHl zqPalqdc55Idomwj>x~VV@aCw)(#( zJFdV7jzdoF%^B(!k5eQ^g#U!Jf5S0LmNJ}T6Dxk(`e1HID$O(g(4Vq7wAABq_*fciv?41tPz_KJp^o#GR7 z&x=UV7o(jLoz@1`46eLenM^JfIp_!x`R|B| zWnx_$h0Kd3c-m;i^`5yyPxf?9iO8Lf>}SglA3j|vM_UNdyevPw(Isrj(R*em?0VVf zZM)xnP}htlCq~r4GfURrcG$gsnuw6vXlnn=!;W|0Z3H<H+^eJ zcWmTcpAXz1MwR+?FUxeJt=e^YQcO;fYv;ERs8#gOENoZqunFEy} zmi}-8syE@KDBC9-;Hd9Wxw&)M0$C|?y9dCHatdV*fx#HzVw!fUp;N-}4qKkTzx7747tEMC($Qo>HGz7l}yW1aeT|ePmR&S%N+%8 z@9rK-mg!vimKCqawpHh(W*vu%rSBP6Bf5&1&a8j(5iySABZ&~BoU8?LAkFUnc9{dn z3qZZZeC6-CzVimTE);|`RDNsUvIRl3Dk!i+@pV1M?2`g2PmQ^+Z*X)IxvSWc-p}u5 z&y3sCClDaLJ3Luv_n7P>GIA>|8P9V$b>{bV<9sFxUilgPe2Q0!hFM)b?3JgN<=HHU z!5dRlJL%{f;bXK=2EkB-%8){_27kDWfI|iGt`MT5`5@4fk5M3PodNP{(>j*pY=d-9 zEoptS;Dhe7H>!`B?f!&}gYFzWrLw^~d1QjXaw&E{H=i$0MoV{WuPYsjTOLLiKBq%* z?xQVz`9vf$6U{-|Wz8!S&Oc1tKL{!m1>pAOXkTx~Nj1HU&=%Y_F!t_MmvA&wOmy5y zKS88rvGZaP`Pm-dt_^*ARX7ADh%|hAgoCM;9$sRD|EJmsI1cYVb+|Wd;~Zb6M?uuP zFnuHa2+FoG9jq%pH~I%fC59ZJM9IhEf3>ehRk?1O4tHv;KzLjq9$W>3dk*cJQcSc# zb$S<#-Ys|b{-2(~lyQJQC<2+-5C(m^o_KYt)1Y>AJQ@LnRQv%vdqNzcNT=Sp^DrGe zZiM4ESt4KF1(GconEnGh1K5oil|C!<1jO^v6V<6si&!W5D20Y0hiA(kK{uqFF6pWJ z0!hRh=TiW8|2KLj-he8rfPk^x|0y<3%=GUx!Te>D*S)utC?igR_3?h2XtBS>{Qdpe z;P{vNel6?};A#=tE_%P5Z%aityNB?ITGjp)@wa1IkAcirj;qm{}VLi=3xf zZ`se|5c|PG@e!y9XmZ#NZ+P8cJ_!OsfPTO-1#bNV%K=G>V)W2>5Dry^V^B|{NV`Qc z-CHUFo)ML?G$D!!z?qN4WrYWV^?_=m0r^&!Bl(NBXi5d*{=i#sjaHNA4I{Yxp*%47 z!s}mPB$Uga`Un&a6x1fR<8A<_L!q-KNm8-^bv|#g%*lD6{SYYek_QN`(qEVbD(~>v ztZ~AznFuLS&M4S`I2)+0g8>3!(LCX>D$kowKwS*Y4;G#?15l-2XaBtnp8fd@4Y^b@ z#?`@0&j68kP|A@TS|O?FKj%`=bOfgiWCW(?JqRiuZjEXMIN)3O@(aYqiO80?4MSI&y9yB z%IuYN*xX1sI~{Krw|>N-_l0A?`7AN=F=;gmuG$tx0iocl;Uq?*v|b|wRF1z`nnLBg zdMjS1m#43|?8awT}sn;V$5nvYBZlQ?~4v%(|avg8>?ApCDRMAjq>?Byl%gFYZ>f!kG-Ay|CoV&w(argswG;Ku7%lqN{l@-~llvP)hsw806Rl!$j~v!P>tw48!Z~W49kCDW9~PS|5x*&a!(S?4 zo(5>1`9wj-+U=0;U2sxcxzu%?5r-PJ2CHMg;#7HhT9Bq!(L<*&nAhX@mEphvCYUl0 zhz&7(R@q4cB5@6gFb1Cn8MSJsKp@OI;1rkxM3yGGNClga{3P`4W`SwSdokx9$oY16 zmz-s~HE+O~pm~BNE2}ktZAn}SzddZlgYH9E#3x6{hsqc8qLPw%HIV&U{;c`N`ezux z^u8pVSe=v?E$%c_csh(wef|1PSIagXYppL5;oPwH3FK2On~f$f3=2{)(X>iL%6^^5 z1=d<4O+p>Cb>Mw<5)Ty;k$*%4g6Ip+b>KX9#2S!wKii+Bb36GNkq(pq6#RICMXnm# z9k_ykU`7G|Er1bEcG(Or3u){=UHa>kZLiV06Q{xVD$#PPMnTs3mIW;YrVPM?ddz z4GGvVIUCJhI*8|)Gkgy>^A9i%euwnF3^|S-pJqMTxw!P7C(-dK2knmX>mSDCU#@oW z*o9&+X)60BpM;QpUe!w7{&?YiPTo7a=n&8r#$@l&Fjs5zRi1ktpCznc% zH8r4FgF&AqlBr^;wqG;>|E{CntKyB!SQUS&S(1RPTQ4^#*QpbS5-P} zikZu&F&VA1hk~IUOT)5nW?z?nJ)cgJ)jjQ7@n+Ixj-(GDy!XuxhHQVZ9Lb9GZ`0O6 z-|r}r%@p;O&7@7zJtq!(zx$&g*?i$?rPbGC|O)G|#hRfL1) zqbaN;2`>m$=l?2O&1N>7MDP^yW@?YawyI(yT0r zs8wWy6neq)I*ZF&iU$r>Nj2IS&zln=Q(IhCTbP&`nyV)gcjG%DyG2iZK8LLf!2{c+ zw?wo7yJx53c`sO7wv?6kv{FxoZv(Cvjx_=m~j208l$oq2J5G@oAX@D6xA5 zm#7kMs0$j9>u9`&(i56(5)k_F zi;skzRCYkF&YO68w@|S}P~^=q9|UKO3A+rE{c_{3D_6{;Uoy2Gj0B%#lusb6=z_#O zTz$Xe@nz~lvwLYnhRo9ivUimI=D@+bhnp!CYt`X`vJ)P+-zZ~*-2T1$O~o~X5q)O> z2Y;c)hR(zJAX+P>(qM{d^J)rm+BpK5=;K84kigTuP+2%o#t|5!$v(G#Fn2%j(EoWv zz1bWtx|n7k;5hdB)E{)3t*-o@YeqQMdBJAMq{A95|JnHYcP9ynE*d*WE5!_a;18V3 zSr)q6N2pSRdsS-cLH zulGM#`vJCV-&3-zC3gs5RM(-=Ami3t3J_|KB+_GQsuqd_`bm8l1`6I{{W4LQy(T5C zSBk~K?Dy;J1obv2xQ3E?0j&m`9%R0#{%@>Sr#CB4b#pL*Y0TZ8rCI7C(jsNnKy~Px z(s7>MXNtE|Mf1Z+Wj$ysrSlbiV3LxWOlcOU$73L}#AgHy;k%k`r>nP9U+3o&k++AF z`&tkWu41fG6~9j2r6cvzm{UzPX6%;i((;j(mV02iyNqi7hs!|cKN+mnDN#!owYf0~mUUST0@cqV@>|`A3i4S1SY${?b zuGP6MDk{t0IJQ>)gc==`nPhQ^KbtnyU{a#*Zn8I5X7XHn0pc=`|iWj400>53>yhN(Xsc{0{AO&Y@mB}8a3?_~%=Qa*(#!HNTS zb~R!VcZ2x0d8Zl*k+0zUvH{NF=z4MJ->{xR$2W)n< z>!)h$^;job>dDOQ$vpH#K_hb0F2DNLrm8n}mOar~NjdqUl0A}H zW#ppD7ss+}y>se-go3+(y~bVGXDrPYLGL9UU9B#O?RbJ~nO62AG&BBN$3F7Z4L zMKZZOArB?HRz^>`0)dH;P5js_!)QBvZB%^_KuaKi?l8 zYdx*z!FljP%nvV}F^{tNXm^fNo>K}xosmwYHjzaq*ByTNi+A^$cY}nT2D^3rnG&Ji zDQ@3Qw(dYJ^au744)%sUK(GelpXJoLn zQ3@RH6qU^*r{8&L3evG*6EVF~$j{!8kTjWB@k=#*vRR!lJpC)m99B;e4mUC-l{>oDc{jbIJ0ON?Et*`RWPsCZO`9@B5Nuaw8yW$4=MltB1UXkW3^ytvrn zGCj+?YnFHxwArHQR!AaoO)szdIWdS55Y=eE*fI<3Hm;1zt1XEeY`1`w;=K1JVDOk7<8vi_o`0cxE`|&lnI_vmdJZ=Ju7YFKQ_x+Ec*xC>{D2t`m zCOsx2Exv*j;zzS=IG5j4Xm~2mvEq}uKqm2d^8+nWkPi63T&{GL#nt9l8_`)~WE3u}-}# zk*z9Md`++WKOn%v(R|aD@zMy@6ndcz5ds-?%DQcaMse*qF-kjSk=Jogd&ECB>NjOG zI}Iizs1AkWJl4rAom$=7E64YA9HmTrp;~M{NSJo<|!LY&C zz#y}pr`_{djmfq;Ydy&6K`KocN^AU`yyF7ZY;=)+6<#>Cqh>#S%lQWJtN3@84v(0X z>x{VFT;*~rC`GT4pEZ7tu@J4U=i-X79&EE>G~$okAy%98@*s7D&;EW>#$)#~RDLdo z$!UAYba$acKeCL;L^-Wv!zDA%>eY}zI^%N^5?B@KP(yj&5()Hc8qG^=|ypH(Dp#GREqc% zMbu>A5tsGM*Yo#eI*`^PzuX?w6W|&Zvn5d5#5H`7rBn*xq~cX}N6gXD40fM>T>|i{ zMv8wPUgsUzF!Lh9%s$xV+My&6IXs-LG!(D3T_CuaQeFi6tbq#JQWQ%MtsD>43S&j$-^5kd->rNw5rUJdXnAr3Pxn`A#8BE~|+{nHTc zL*e)i1~&imNu=!5?Mz%9(xQGzk$~oX3fApA>WClMNl^I^A)5ZnW$tWT>LOPr6?zs` z&b@Qa9(IO_WjX;KRu__KT6L!*!<)tObDkuVRIEWhz8RTs_YZD5dRIF3%@T^*kek|ODzSD|ieHXJ7TDS`<~IX7PC zRM8vgBcthJl^cWaT`RsnVSL*#thkz}y!4&|&z3ZpY2_CnRdd2cX|(=SCdP_8%YaNM zf@8F|C3+`zlya$irHDWi`=QYWTi-W$%w;V|oyV5whjZve61@`DZ=RPrHKDW<-@eJ| z%j^*H@~&~@kKOt9yh>?22ITOX*RMt=Tq?|L+P((9Qx+YEY!g_rQzIE}7Ea3(&chQ1 z20f4$%XWTQ$l{o5%~+&dFka=wX*Q6#+)KES!D&t*myc1X(Z+a$&l2IJ%4nH4Qmb=d ztaM59BIIWU08&}mPD8maOBs&v7&C7UUJ25;=%z`Xp7Mw7QfAN!^iP^AD1nNmpDb@) z&L%LZyX^{B>Tg=#N-`~`OARHvX5sX@x?qPXFk|}P6Y9?a>17*%pnYGDF9qM9fM5|r zI#)w7`7;x&540e-w571hv53z>DY)nh-#(~Un%wGZpnI^1=6)4uxBkoTPAz0JM?(*0eK^SB^vh9~|E(g(W zmfvEt0CE+nzM7w)^>kqobfLz*BAlw`;l=T<$<^<$GQ<}>v^@Y`cX@)7C!QTKP+J{p zqoa8_OSP_2MfrLFtWzVDUb{%$0$A1nAJ}KvyEH;tK*a=9qVfsyL*sCs_p87PerBOW z6DICup_WO^goNfVH5v{iMg#@#cp)^a%@Q=ac;ZT|lCSVSEc!3`uIA)NY_e&EDx1q2 zJ9Izow{B1xe;Uz-xxbN)~Pq#AqX(gbc4LKhxu=FlAmGQ~5I>pkScB@B=ihd?#|q5ZT93Tn;}*zM{q84+ z9(S#uz1Ei)zvlaPmDD4Edk1z)D`7fRYT~2l_bMG*h~fAw`DI>4j_3+bB~QAH{T}BP zSS%{jN^fX!NAoI>fg16rtwznVB5Z6bCE6rm%1^#To#Af1~e{1Xe(tO*&Yg*NfY_aO*>L7b8*;H#Bz`r-w*tt^b2 z775_9O6wDZ@&>Xzf`|MeqwZweijOLbZ8{o~X4}Dp4lkwrP9!fzCuF5?{_;mzX~qUU z73L!cisc-Bez;;i3LZ7G_uyech%z(XMSH(n>6=+<){ja0PGhv^4PCjW$vk&s=t9CV z;RusEiygg6roR#@?kAQbUdD~RA8jSP7LQ9m9Jy0{76hP5InU>xil6!35S1ikAUUE? zPu9hAPgQHIIIs4*&v9n}vmx=t8`^Z4;{b?kAbX~Q z{<TV#C7Oy<^(17(G2U#$#fQ9)Xsj+H5! zPg(v@RM;jM_{8?3Jv7#=xRZ#3B;R<$Ic2Ko;ozk%UN_ZyG^AIYKG7YNj@-tNG>t7T z=eXJ|R=)&C_5E3~#SWP%N7Ws_{J<9Jc@N&}!YqAw*{&j8sGRAiB^?faNz zgwD0|yH=-C=+s$D^_rQ3aERrx(?LB&B|ESS0_3kd_abA(h$eEnHqkq(CB4CRl~^>7 zN*)BYYw@bN#{7$={ga?&hoR*0$Wi}`M}q(!&Bgz@8R7pr1{2RyCKiI}O8xsmq3D#O zz+@j7|DAnC=yrst*X|W&lKv`&PI_cZjVSvXV81(!*$lC?SdOfWs90p86MmDj!}+S% z4`jv7lGE2gWz2nEgAWJv`Xkr8(iXJ08bkuQm$kcdS0S&)vLe!jEa>Asqc8fHW zt!p>$L$=yRvBvFY>XwQr^VRRnc8z+w$io`amn)Z)!d^Mr3z*L*E&~3|4v!X4`+9I)V0BHE4(f_uu~ZsDW2w3>`=*(s$omo}SG2bl#9Ex0!oZsxFGL zGlMRP)*3~za&>|#aPm8h)HFOxETKAy!#lYB2r%MG8JC1;`uG))!g!tnW9?_STLmYoIB}PmTc}V|E47YMe zyTuWMSJXLKua%mVJpOr`PuLraHwBQ_giPfXW4Vj!eD2#+mrODX+sB(@wlCi{HZ^6v zDdPCK@H>&wn#!38p;Z zK2#T0_0?ziC@oLg-xDLcEpYn&ocyny&_lG~p17<~&;Pbx@Cj7q+|1EjM1FmE)HT&q zqgZ4q9HIDnPP9P5s<*6OtA&n4JY}W~3o9?;hX1a^)u%`~zo+V33%lzHA6L1qg7)%^ z%@U%fw#V%eZJz85`b2!eXols60<(K}I}|1P%Z+W11s*`!mr(vdt$9i{QTS_2#D|0B z;^B#~-rjQazI>A98-4l~b*;Z&A&ze4wi_kO)bPGs4j47iI0Zd5E2LglQtDH)ozG-( zcYZ26j=%e(0mbN)gmZ%z@5GBdbLnaYqSNfAnlbuG5bF)Cf?=yhQFbpjo1M-L|DrT` zj>KxVMvW^jN8V^=LRVki4)O6&SGRN&sIv0Mu$&~C?6@t>6Co(c9QdWX;J7=5>Z_w; z`|?#BM}i3E?~eAgmghWWq1QsLIG0hxAA3vGcNkX3jyUg&mqGT76sU!WNk878KU10+ za#ev|#WK#aq0HJzS>;*a59&YK3&ZW+3Yy6qh5Fx;m+d8{V3qRIXWxnZRh_t@z0% z$aWZ6RJ3a;k=||Yds%5^JlF&)V#i-=o@s&Q245h(%Krqon6T(b9K(4^Aek?Q1evLU z0bzP%F4gn{wQ6|0M=p>1Krn_Q(4;&p#hA9Zf|7?ieGs59{B+d&L5AjlSfYpft3$?2LD1$r$mxZJE+q#ul%E zO=h;d54Uc8c5(5EYl(_9e;tv8gk44+RL!_w&)$^CW!4%=QVEyV@w&EX-iRK0AuBn9 zEu!+7W1h?2w3c8f<%>;rYEfMnr`GuF*$77z9kLMn024;he zg!O-N!=2Ayo-D%AjO~AUvX}oa&+2r%M{R-Vk|0cy6F}0&0D<{9>3J59_!16i9OK?U zVgziDt1}i*MqJcA_>h$E5uiz>fC!}`kim>+QpW_TQ6~!GT}D(sR+#S;Jlh5Gvi=Cz z(FZmFJCWY|(F7wSYwZ?^)ys9d0L=0Xs8*(mz9j<*;OBsTR06V(yUQKR0b7+NG%Frp z^(nBFRMQU(9aG8xsJ9&)9lhxP_yi`v2k5NQw;<8Q6FU&kfNS_ghI7g1*Umf zj|Rj#Cz3)?K+CESkf{F7nYfDzo3Dc?C|>IVh9!pAUQegZOI7?DCrpA6vB-E%eO&ur(pOn{o!RC5xLH8Z$F)YFSCltVig7POh(f$X^ypw%hM4O6(SgMsFk^&fVn@>C>Xt-HZse6$cT z@&LP_s774t@x8;}biWk%iBHl4lqxlwDJNTx7`5w5-H?UB_D9Xs@M~cAyWkm$T|K6& zj|HvJ5OLI5Lfv=Bmk=egYHbOc*a*ZpUjt>W7>eicoHGN(um&(U;Qg4kJ_a;+jhOla zr!&WJkUpVq`o$kRox17#DqZku+$8)zi|txeepmS~`om#I0jBpWb3gz#+`y#L*x1-P zI?6Uw=6!R?otl=GqmvH151TmHI432n4~Io%Utfq_pUb5S5oy=kgsJ!fB|(^GSnZvd zBc(Wad!f3(6#L2#P3&P8rFZ~wJblw(rwa&w?frB7kg?UU#-LWpDDr-L9%ZeFB9mA|j#zAcbMt zt6qQhB7xS}I*@D=uzDFC0fh2s9x;I#BLcsBKpr*@#m*4{HDAc$jkB@79wj<6(@D&2 zB}NqxsyYUU3$PI+3B=7DPIpvcs%dt1cB|?`HL%A8QyyHzd_r=s4IM)l4TyQ$Z9&%^ zJTQi{6NFKbVB1W-%$Qk=i;L9y&tUR(-|-=rklK<}BV*W9)hUE%QVgu~eY9@-8U_&1 zpjpuX8gVYAhi3M$@4=&CeEZzt00945@UD@9wTU9T(8g*nSwJ&cD%9+$6@O$h%Vjf$l^lOg zvYA*6Az%_u= z7$W2AV1>25KWG3q&~g704~+pwa&%0L;Q+C@nHd!~H~t%GJG@N)Dx^l7(5eh9YiV#A z#b^JJ&m=fFI22up*t~>w#i3E^wA5;muGw%%)eqR=zP{_ygB1ogbW`y$ z#0^3y73InBw;%ow(b(Z?yF&UHm0{h2-Ace0(3&}d?D);!BM=fi12~I7tVq@T`+J35 z_B}S}|6}1_dH)^Zq3_h}^ZdmQy>XvZpM;$s2j)ErZD}bqd%F%`LBsy2u2(n@>dW%N! zB0ub#!lE;8Nz4c=$6hVeZjsL#8W_DfdcDx{md5Y$V$uW?kc+wf^du7k(w(>~i^D}f_)e^mm_83sA(f5u&V_V^+ zVl?ZfeNO+@x)Qb2T40pOl&87E6oC;%l5&6v9BK3e3W7ZT<=EpBodEyaGr)%K%qzl*go zL)y4c+D~mKnDK7(i2UhP(6fD;I12Z)Nm|z2pgzOuWj)lVzNZ&X35+Y#y!z5!UIN63 z++B6h*@K&an?jP6ihX|frOm=^b@c+*LzmgIh*{-wXf@5ZE%)W;EO%a)URQJilowXj z?fq5jhZd7nZ6f<>b27`+EWhpC_v(K?vv^EeSTS9wjxwxPK5%?hYduL2Rc9NbC+z*@ zxhsPD$r^g)G2*%;+D~yX)8osuOk<6B0Es`aeFNJKJBCgs`(x_$YIC@`e`edp5vf!~!4!3J_JT-mb~ z{c`wUZ=eMkl`g$b5#H|1RO#r_yXkzs3aBUPs94>j9oO&hKO|CEeShGKWjP+tdRV#X z4O|M=GZNk$&lD-{8I#UxXi8qyq@YS;Mq4a(tk_WRJeCIs&@zYq3C?uK<}Gj29>Xo4-Rcgwm@ zVLgM$*FoN4xEtQ7jTF>PSl6Cy~`=r70uiMOtS)JuyqV`(%m+-F>kQZ2vp9JHi{) z;a^ye%R2__)&hHAI=UbnxLT&VV2k31&5-e40shDj>z=2g{iwE^uw`7V{sX=Ac{?Fd zs#8OpBYmsdkeDVXlw*#(uhy01Qnh4%d~i@pt1= z`3lM5o;AM;37Vx6xjcU(@>pg=8@Hh>0?974!{=R)Kx8T|K8Xxml5ONqEAsW1Rz5$; zZDaMDKfmR3fV-NO=JAZgYYke8?r1VNuUq(jpC4OS z7^$8R(>zU|(JfjvG!&??UleFAnH^umZ!vb;vHPV|oS@iu`Fs9J$>fihSN4PscD(U} ztm4ZaxXSsy*-OqIh!e3}d%yBqa-dVikr#{k6t?1eF{zMlR8Q;H+t2U_) z{W_7L+0UUVskLb2W+RQHpfK>&q$wogPsy+KoJryNR{ovY4z^ zV>SC*jEq-TmGoQ}r7~Y{3|+R?@|7<%LPW`g!@b(uo8>`u)1}r7`NU0HtWyK`n~|0x zis~3IY?rp?cFj1)4EVeko#my(W?8q=m4&>>{*o+Op+}G52;fHxHHqMePX{aYU~b>$ zMW+^oq91O)XWg9kzvw`092JBD0y2pb|AsBF6a68XazanlVu4Nn=V97fK4mN74(0!T??oOCY*09L>4wRHMD$GWgTFSIq zWh-!f_C1de(Qhud@|H387DKwkm&W#|WCV_DE}W`vcm`CLb}-2lnl7d*l~|nkRz+=4v%UYgh#Rs(@a+`G zd80(-%UBGx!2p5Wliyp!3Q5!UHQ<@dUHJuu&m zE1H6#K3o0x&)#=X-|+ixJ@Aj~BSLPUATnG&r>_>eBZuPBCn??J3K_vUhy4!MJ*bPe z%uJy_T0L=L@~{16+Vy?84CJfbQ3VD=J+C}+rQ|TkE(y}3b|c%Ccl(e|-)!&uos*g0 zx*{W7w&m(0E-IEqwK@zM-K<$e9qEj)EFi2xkY0;{=@B*aI1y$^M>x?0Vsq8y7*zeQ zUUjVPOqW!Da4O$8DJHA#+b=EL_M*kslrsSmJ{qkymg$g{Skqy}0ykS3cbjmnDhaLM zJT~)%eRytxi2e&x#9bV26`B@>diW;$f37OI%|7qF!&K58#o3)M)YGl_=1t2_6iLDz zTuPZ5?eTh+jfLf*_KdB6hP|*abDXLPhs?wU4L55HBvWb?3oSkq4xtmLXR{18EPg(X zPLD}nu>6`XqTDc+%r=zLTy6Rgk4f$1%U}?qbF9u=I(<2*-&%FdvtK#G;#3{xnk8ix zjsDc3ox$*lPgke?ktCKHEv#A%I}gdui$~l+Q4pVifRXtwtM9IE9@^z;muu&2Fv; z@hhJwL3e`@L$=mMFDYlE`uer|c5M}*2*XdUFXrdwin!lpbd@hw!Q#XD25;FY$yC9& zWF{=jm3E^$wQqtWXj6p@seX;7fB0zCnq5=FX*KO5q2JIjdYeIb!HRROdfI-q=t6Z1 zUfF9|iYw79TD6Dw9fCvp3ZK=1;=tiaRZ(VJGUHJ-i|!?)FZS|L3K#nOnmF92kr~nI zJhJCke2s+$ER~+hTxcNBRw#fvky$UuZ+gqSBFGSy7leeQw~kl&dQYQyAS5uUPOt>U zZUw%LnAL4=hdg|AV!?TGb3A--SYE7+f7tUVY~-cVMQ*8;h{)Et2a1e9GHJEVhSQ?t zk784qTFbfD`f*kMpDJ{oSC9#x9)iGGvB}J}_3Tr?`Bvbo#Oohfa7cS>2I@0J#TVDN zsK?3gK$zQAnw$#BQ0joj!@wS?Qkz zG1&_Uw8xJ)96Kc7R!>$f{OI8cz9#rD{H*NzX2#|> z&zQeYR{X(tetxYwG8?daZ7+iUX+{+PdX8T(U>$O7X+HMT4Kw1`!%GVd7T6_&@!9!{ zJvxWFzdVK`-aZt(S-N5R);{u)ebC;HhlQd`2*i7^d+H2COHyH`Ah)*9m#x#hddhs) zg~ah@GCC&KjW^JXE7_f6itfpZv6q8HU~Vnb`UAn)YUHAaCC~5QzZ3y`(V<`SMnw|w zTkvNNDk#@So~)=LM`ww2*e7p0l^VGszca(>zO9e5yprGPMimbBl`4G<+xF;wQ0PBO zv)I;q&B%xBAAEssAo+JQBvb70fdhuYA>T3U-#A0DlN}xd?%^@L2KL|QMXx9X4t?lP zb_avy-=@J}XTl;$x-0zt)?kMN)?k|zLL;r{f6u{VkAOoE>lQ*&3;O$9(eR==-@;+C X Date: Thu, 25 Aug 2022 16:58:20 -0700 Subject: [PATCH 35/57] added error messages page and link fixes to a few pages --- website/content/docs/nia/usage/errors-ref.mdx | 81 +++++++++++++++++++ .../content/docs/nia/usage/requirements.mdx | 2 +- website/content/docs/nia/usage/run-ha.mdx | 2 +- website/data/docs-nav-data.json | 4 + 4 files changed, 87 insertions(+), 2 deletions(-) create mode 100644 website/content/docs/nia/usage/errors-ref.mdx diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx new file mode 100644 index 0000000000..c4aa3daa7b --- /dev/null +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -0,0 +1,81 @@ +--- +layout: docs +page_title: Error Messages +description: >- + Look up Consul-Terraform-Sync error message to learn how to resolve potential issues using CTS. +--- + +# Error Messages + +This topic explains error messages you may encounter when using Consul-Terraform-Sync (CTS). + +## Example error log messages + +If you configured the CTS cluster to run in [high availability mode](/docs/nia/usage/run-ha) and the the local module is missing, then the following message appears in the log: + +```shell-session +[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" +``` + +The resolution is to create the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. + +## Example API and CLI error messages + +In the following example response error: + +* CTS can determine the leader. +* `high_availability.instance.address` is configured for the leader. +* The CTS instance you sent the request to is not the leader. + + +```json +{ + "error": { + "message": "redirect requests to leader 'cts-01' at cts-01.example.com:8558" + } +} +``` + +The resolution is to redirect the request to the leader instance, for example: + +```shell-session +$ curl --request GET cts-01.example.com:8558/v1/tasks +``` +--- +In the following example response error: + +* CTS can determine the leader. +* The CTS instance you sent the request to is not the leader. +* `high_availability.instance.address` is not configured. + +```json + +{ + "error": { + "message": "redirect requests to leader 'cts-01'" + } +} +``` +The resolution is to identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. + +--- + +In the following example response error: + +* The CTS instance you sent the request to is not the leader. +* The CTS is unable to determine the leader. +* Note that these conditions are rare. + +```json +{ + "error": { + "message": "redirect requests to leader" + } +} +``` +The resolution is to identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: + +```shell-session +[INFO] ha: acquired leadership lock: id= +``` + diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx index 31e8ac186f..46c20da43c 100644 --- a/website/content/docs/nia/usage/requirements.mdx +++ b/website/content/docs/nia/usage/requirements.mdx @@ -89,7 +89,7 @@ The Terraform module for a task in CTS is the core component of the integration. Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. -Refer to [Configuration](/docs/nia/installation/configure) for information about configuring CTS and how to use Terraform providers and modules for tasks. +Refer to [Configuration](/docs/nia/configuration) for information about configuring CTS and how to use Terraform providers and modules for tasks. ### Partner Terraform Modules diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 50e8dda098..7f8da2a750 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -84,7 +84,7 @@ If the `high_availability.cluster.storage.namespace` field is configured, then y We recommend deploying a cluster that includes three CTS instances. This is so that the cluster has one leader and two followers. -1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/installation/configure) for all configuration options. +1. Create an HCL configuration file that includes the settings you want to include, including the `high_availability` block. Refer to [Configuration Options for Consul-Terraform-Sync](/docs/nia/configuration) for all configuration options. 1. Issue the startup command and pass the configuration file. Refer to the [`start` command reference](/docs/nia/cli/start#modes) for additional information about CTS startup modes. ```shell-session $ consul-terraform-sync start -config-file ha-config.hcl diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 224d5184f7..90156e9bcc 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -813,6 +813,10 @@ { "title": "Run Consul-Terraform-Sync with High Availability", "path": "nia/usage/run-ha" + }, + { + "title": "Error Messages", + "path": "nia/usage/errors-ref" } ] }, From 54be860069a3a37b70f3e09e14d7619661f9f1a7 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 17:18:06 -0700 Subject: [PATCH 36/57] updates to the architecture page --- website/content/docs/nia/architecture.mdx | 67 +++++++++++++++-------- 1 file changed, 43 insertions(+), 24 deletions(-) diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index a6712a9733..b112b3a1b8 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -2,37 +2,29 @@ layout: docs page_title: Architecture description: >- - Consul-Terraform-Sync Architecture + Learn about the Consul-Terraform-Sync architecture and high-level CTS components, such as the Terraform driver and tasks. --- # Consul-Terraform-Sync Architecture -Consul-Terraform-Sync (CTS) is a service-oriented tool for managing -network infrastructure near real-time. CTS runs as a daemon -and integrates the network topology maintained by your Consul cluster with your -network infrastructure to dynamically secure and connect services. +Consul-Terraform-Sync (CTS) is a service-oriented tool for managing network infrastructure near real-time. CTS runs as a daemon and integrates the network topology maintained by your Consul cluster with your network infrastructure to dynamically secure and connect services. -## 10,000 Foot View +## CTS workflow + +The following diagram shows the CTS workflow as it monitors the Consul service catalog for updates. [![Consul-Terraform-Sync Architecture](/img/nia-highlevel-diagram.svg)](/img/nia-highlevel-diagram.svg) -The diagram shows CTS monitoring the Consul service catalog -for updates and utilizing Terraform to update the state of the infrastructure. +1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and Views](#watcher-and-views). +1. CTS detects a change. +1. CTS prompts Terraform to update the state of the infrastructure. -There are two principal aspects of Sync to know about corresponding to the -lines to Consul and Terraform in the diagram above. The line to Consul -represents the Watchers monitoring the state of Consul's service catalog (and -possibly KV store) while the line to Terraform represents tasks being run to -update the infrastructure. -## Watcher and Views +## Watcher and views -CTS monitors Consul for updates utilizing Consul's [Blocking -Queries](/api-docs/features/blocking) whenever supported, falling back on -polling when not. The watcher maintains a separate thread (known internally as -a view) for each value monitored, running any tasks that depend on that watched -value whenever it's updated. Say, for example, running a task to update a -proxy when an instance goes unhealthy. +CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as watchers. + +The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it's updated. These threads are referred to as views. For example, a thread may run a task to update a proxy when the watcher sees that an instance changes to an unhealthy state. ## Tasks @@ -53,8 +45,35 @@ network infrastructure. The following [drivers](/docs/nia/network-drivers#terraf Each driver includes a set of providers that [enables support](/docs/nia/terraform-modules) for a wide variety of infrastructure applications. -## Security Guidelines +## State storage and persistence -The [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) -tutorial contains a checklist of best practices to secure your -CTS installation for a production environment. +The following types of state information are associated with CTS. + +### Terraform state information + +By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). If the backend is not configured to a local location, then the data persists if CTS stops. + +### CTS task and event data + +By default, CTS stores task and event data in the Consul KV. This data is transient and does not persist unless you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha). High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. + +The data stored when operating in high availability mode includes task changes made using the task API or CLI, such as creating a new task, deleting a task, or enabling/disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). + +## Instance compatibility checks (high availability) + +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently, enabling CTS to properly perform automations configured in the state storage. + +The only incompatibility CTS checks for are tasks that are configured with a [local module](/docs/nia/configuration#module). CTS instances that do not include this module directory are incompatible. Example log: + +```shell-session +[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" +``` +Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information. + +CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. + +CTS will continue to run and log the error message if it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility will not run successfully. This is because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. + +## Security guidelines + +We recommend following the network security guidelines described in the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS) tutorial. The tutorial contains a checklist of best practices to secure your CTS installation for a production environment. From af8f00b3eb2ad7f789b74f32a5e383c54678253b Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 17:59:49 -0700 Subject: [PATCH 37/57] added ha information to the API endpoints docs --- website/content/docs/nia/api/status.mdx | 4 ++-- website/content/docs/nia/api/tasks.mdx | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index edfd512072..a129aa3215 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -6,9 +6,9 @@ description: >- --- # Status -The `/status` endpoints share status-related information for tasks. This information is available for understanding the status of individual tasks and across tasks. +The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). -The health status value is determined by aggregating the success or failure of the event of a task detecting changes in Consul services and then updating network infrastructure. Currently, only the 5 most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Overall Status diff --git a/website/content/docs/nia/api/tasks.mdx b/website/content/docs/nia/api/tasks.mdx index b870d26718..0e871224d8 100644 --- a/website/content/docs/nia/api/tasks.mdx +++ b/website/content/docs/nia/api/tasks.mdx @@ -9,6 +9,8 @@ description: >- The `/tasks` endpoints interact with the tasks that Consul-Terraform-Sync (CTS) is responsible for running. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. + ## Get Tasks This endpoint returns information about all existing tasks. From b170a3c13b1a45449bb3de520802ecf8d9409ef3 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 19:34:22 -0700 Subject: [PATCH 38/57] add HA to configuration reference, updated other configuration params --- website/content/docs/nia/configuration.mdx | 58 +++++++++++++++++----- 1 file changed, 45 insertions(+), 13 deletions(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index d57dde75b8..7b665205d6 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -5,11 +5,11 @@ description: >- Consul-Terraform-Sync requires a Terraform Provider, a Terraform Module and a running Consul Cluster outside of the consul-terraform-sync daemon. --- -# Configuration Options for Consul-Terraform-Sync +# Consul-Terraform-Sync Configuration -The Consul-Terraform-Sync (CTS) daemon is configured using configuration files and supports [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) (HCL) and JSON file formats. +This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL and JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. -## Global Config Options +## Global configurations Top level options are reserved for configuring CTS. @@ -44,7 +44,7 @@ tls { - `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - `log_level` - (string: "INFO") The log level to use for CTS logging. This defaults to "INFO". The available log levels are "TRACE", "DEBUG", "INFO", "WARN", and "ERR". - `port` - (int: 8558) The port for CTS to use to serve API requests. -- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. Used as the service ID for CTS if service registration is enabled. +- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in the cluster. - `syslog` - Specifies the syslog server for logging. - `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging. - `facility` - (string: "local0") Name of the syslog facility to log to. @@ -82,7 +82,7 @@ license { | `auto_retrieval` | Optional | object | Configures the license auto-retrieval used by CTS. To learn more, review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for details | Review [Auto-Retrieval](/docs/nia/configuration#auto-retrieval) for defaults. | -#### Auto-Retrieval +### Auto-retrieval You can use the `auto_retrieval` block to configure the the automatic license retrieval in CTS. When enabled, CTS attempts to retrieve a new license from its configured Consul Enterprise backend once a day. If CTS cannot retrieve a license and the current license is reaching its expiration date, CTS attempts to retrieve a license with increased frequency, as defined by the [License Expiration Date Handling](/docs/nia/enterprise/license#license-expiration-handling). @@ -92,7 +92,7 @@ You can use the `auto_retrieval` block to configure the the automatic license re | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | string | If set to true, enables license auto-retrieval | true | -## Consul +## Consul connection The `consul` block configures the CTS connection with a Consul agent so that CTS can perform queries for task execution. It also configures the automatic registration of CTS as a service with Consul. @@ -127,7 +127,7 @@ consul { | `transport` | Optional | [transport](/docs/nia/configuration#transport) | Low-level network connection details || | `service_registration` | Optional| [service_registration](/docs/nia/configuration#service-registration) | Options for how CTS will register itself as a service with a health check to Consul. || -##### ACL Requirements +### ACL requirements The following table describes the ACL policies required by CTS. For more information, refer to the [Secure Consul-Terraform-Sync for Production](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync) tutorial. | Policy | Resources | @@ -140,7 +140,7 @@ The following table describes the ACL policies required by CTS. For more informa | `keys:write` | `consul-terraform-sync/` Only required when using Consul as the [Terraform backend](/docs/nia/configuration#backend). | -#### Auth +### Auth Configures HTTP basic authentication for communicating with Consul. | Parameter | Required | Type | Description | Default | @@ -149,7 +149,8 @@ Configures HTTP basic authentication for communicating with Consul. | `username` | Optional | string | Username for authentication| none | | `password` | Optional | string | Password for authentication| none | -#### TLS +### TLS + Configure TLS to use a secure client connection with Consul. Using HTTP/2 can solve issues related to hitting Consul's maximum connection limits, as well as improve efficiency when processing many blocking queries. This option is required for Consul-Terraform-Sync when connecting to a [Consul agent with TLS verification enabled for HTTPS connections](/docs/agent/config/config-files#verify_incoming). If Consul is using a self-signed certificate that you have not added to the global CA chain, you can set this certificate with `ca_cert` or `ca_path`. Alternatively, you can disable SSL verification by setting `verify` to false. However, disabling verification is a potential security vulnerability. @@ -164,7 +165,7 @@ If Consul is using a self-signed certificate that you have not added to the glob | `key` | Optional | string | The path to the PEM-encoded private key file used with the client certificate configured by `cert`. Required if Consul has `verify_incoming` set to true.

Can also be provided through the `CONSUL_CLIENT_KEY` environment variable. | none | | `server_name` | Optional | string | The server name to use as the Server Name Indication (SNI) for Consul when connecting via TLS.

Can also be provided through the `CONSUL_TLS_SERVER_NAME` environment variable. | none | -#### Transport +### Transport Configures the low-level network connection details to Consul. To achieve the shortest latency between a Consul service update to a task execution, configure `max_idle_conns_per_host` equal to or greater than the number of services in automation across all tasks. This value should be lower than the configured [`http_max_conns_per_client`](/docs/agent/config/config-files#http_max_conns_per_client) for the Consul agent. @@ -182,15 +183,16 @@ If `max_idle_conns_per_host` and the number of services in automation is greater | `tls_handshake_timeout` | Optional | string | The amount of time to wait to complete the TLS handshake. | `10s`| -#### Service Registration +### Service registration CTS automatically registers itself with Consul as a service with a health check, using the [`id`](/docs/nia/configuration#id) configuration as the service ID. CTS deregisters itself with Consul when CTS stops gracefully. If CTS is unable to register with Consul, then it will log the error and continue without exiting. Service registration requires that the [Consul token](/docs/nia/configuration#consul) has an ACL policy of `service:write` for the CTS service. | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | -| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul.| `true` | -| `service_name` | Optional | string | The service name for CTS. | `consul-terraform-sync` | +| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. +| `true` | +| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high_availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | @@ -213,6 +215,35 @@ The default health check is an [HTTP check](/docs/discovery/checks#http-interval | Timeout | `2s` | | TLSSkipVerify | `false` | +## High availability + +Add a `high_availability` block to your configuration to enable CTS to run in high availability mode. Refer to [Run Consul-Terrform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. The `high_availability` block contains the following configuration items. + +### `high_availability.cluster` + +The `cluster` parameter contains configurations for the cluster you want to operate with high availability enabled. You can configure the following options: + +| Parameter | Description| Required | Type | +| --------- | ---------- | -------- | ------| +| `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information | Optional | Object | + +#### High availability cluster storage + +The `high_availability.cluster.storage` object contains the following configurations. + +| Parameter | Description| Required | Type | +| --------- | ---------- | -------- | ------| +| `parent_path` | Defines a parent path in the Consul KV for CTS to store state information. Default is `consul-terraform-sync/`. CTS automatically appends the cluster name to the parent path, so the effective default directory for state information is `consul-terraform-sync/`. | Optional | String | +| `namespace` | specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | +| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | `15s` | + +### `high_availbility.instance` + +The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: +* `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. + + ## Service ~> Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. See [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. @@ -624,6 +655,7 @@ driver "terraform" { - Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html). Visit the Terraform documentation links for details on backend configuration options. - If omitted, CTS will generate default values and use configurations from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html), which stores Terraform statefiles in the Consul KV. The [ACL token provided for Consul authentication](#consul) is used to read and write to the KV store and requires [Consul KV privileges](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifier appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. - The remote enhanced backend is not supported with the Terraform driver to run operations in Terraform Cloud. Use the [Terraform Cloud driver](#terraform-cloud-driver) to integrate CTS with Terraform Cloud for remote workspaces and remote operations. + - The `local` backend type is not compatible with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. Refer to [Instance compatibility checks](/docs/nia/architecture#instance-compatibility-checks-high-availability) for additional information. - `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the CTS log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. - `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the CTS daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path. - `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. From 61a71ed2bbc9f3a0e43de8f64431781a347996ca Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 25 Aug 2022 19:36:54 -0700 Subject: [PATCH 39/57] updated the compatibility table for 0.7 --- website/content/docs/nia/compatibility.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/website/content/docs/nia/compatibility.mdx b/website/content/docs/nia/compatibility.mdx index c4cc632858..4443085af6 100644 --- a/website/content/docs/nia/compatibility.mdx +++ b/website/content/docs/nia/compatibility.mdx @@ -7,7 +7,7 @@ description: >- # Compatibility -The following tables list the Consul-Terraform-Sync (CTS) version compatibility for Consul, Terraform, and Vault. +This topic describes Consul-Terraform-Sync (CTS) cross-compatibility with Consul, Terraform, and Vault. ## Consul @@ -28,6 +28,7 @@ CTS integration with Terraform is supported for the following: | :------------------ | :-------------------- | :---------------------- | :--------------------------- | | CTS Enterprise 0.4+ | 0.13 - 1.1 | Latest | v202010-2 - Latest | | CTS Enterprise 0.3 | 0.13 - 1.1 | N/A | v202010-2 - Latest | +| CTS OSS 0.7 | 0.13 - 1.2 | N/A | N/A | | CTS OSS 0.3+ | 0.13 - 1.1 | N/A | N/A | | CTS OSS 0.2 | 0.13 - 1.0 | N/A | N/A | | CTS OSS 0.1 | 0.13 - 0.14 | N/A | N/A | From 03ff1d07ef17c9818b0482c0e63f9acb4b48a564 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Fri, 26 Aug 2022 08:38:51 -0700 Subject: [PATCH 40/57] replaced HA diagrams with SVG files --- website/content/docs/nia/usage/run-ha.mdx | 4 ++-- website/public/img/nia/cts-ha-after.svg | 1 + website/public/img/nia/cts-ha-before.svg | 1 + 3 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 website/public/img/nia/cts-ha-after.svg create mode 100644 website/public/img/nia/cts-ha-before.svg diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 7f8da2a750..ef4b46eca0 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -21,11 +21,11 @@ In a standard configuration, CTS exits if errors occur when the CTS instance run The following diagram shows operating state when high availability is enabled: -![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.png) +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.svg) The following diagram shows the CTS cluster state after the leader stops. CTS Instance B becomes the leader responsible for monitoring and running tasks. -![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.png) +![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-after.svg) ### Failover details diff --git a/website/public/img/nia/cts-ha-after.svg b/website/public/img/nia/cts-ha-after.svg new file mode 100644 index 0000000000..088089d205 --- /dev/null +++ b/website/public/img/nia/cts-ha-after.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/website/public/img/nia/cts-ha-before.svg b/website/public/img/nia/cts-ha-before.svg new file mode 100644 index 0000000000..546cf54893 --- /dev/null +++ b/website/public/img/nia/cts-ha-before.svg @@ -0,0 +1 @@ + \ No newline at end of file From cd88085423b9cff93ecac472d1c6fe9f00545518 Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 11:58:19 -0700 Subject: [PATCH 41/57] Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/api/status.mdx | 2 +- website/content/docs/nia/api/tasks.mdx | 2 +- website/content/docs/nia/architecture.mdx | 14 ++++---- website/content/docs/nia/cli/start.mdx | 4 +-- website/content/docs/nia/configuration.mdx | 12 +++---- website/content/docs/nia/usage/errors-ref.mdx | 6 ++-- .../content/docs/nia/usage/requirements.mdx | 30 ++++++++-------- website/content/docs/nia/usage/run-ha.mdx | 34 +++++++------------ website/content/docs/nia/usage/run.mdx | 2 +- 9 files changed, 49 insertions(+), 57 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index a129aa3215..d364f93bec 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -8,7 +8,7 @@ description: >- The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Overall Status diff --git a/website/content/docs/nia/api/tasks.mdx b/website/content/docs/nia/api/tasks.mdx index 0e871224d8..60ca787a3b 100644 --- a/website/content/docs/nia/api/tasks.mdx +++ b/website/content/docs/nia/api/tasks.mdx @@ -9,7 +9,7 @@ description: >- The `/tasks` endpoints interact with the tasks that Consul-Terraform-Sync (CTS) is responsible for running. -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances, but requests to a follower instance return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/tasks` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Get Tasks diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index b112b3a1b8..89514908e5 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -15,16 +15,16 @@ The following diagram shows the CTS workflow as it monitors the Consul service c [![Consul-Terraform-Sync Architecture](/img/nia-highlevel-diagram.svg)](/img/nia-highlevel-diagram.svg) -1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and Views](#watcher-and-views). +1. CTS monitors the state of Consul’s service catalog and its KV store. This process is described in [Watcher and views](#watcher-and-views). 1. CTS detects a change. 1. CTS prompts Terraform to update the state of the infrastructure. ## Watcher and views -CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as watchers. +CTS uses Consul’s [blocking queries](/api-docs/features/blocking) functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes. These mechanisms are referred to in CTS as *watchers*. -The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it's updated. These threads are referred to as views. For example, a thread may run a task to update a proxy when the watcher sees that an instance changes to an unhealthy state. +The watcher maintains a separate thread for each value monitored and runs any tasks that depend on the watched value whenever it is updated. These threads are referred to as _views_. For example, a thread may run a task to update a proxy when the watcher detects that an instance has become unhealthy . ## Tasks @@ -51,17 +51,17 @@ The following types of state information are associated with CTS. ### Terraform state information -By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). If the backend is not configured to a local location, then the data persists if CTS stops. +By default, CTS stores [Terraform state data](https://www.terraform.io/docs/state/index.html) in the Consul KV, but you can specify where this information is stored by configuring the `backend` setting in the [Terraform driver configuration](/docs/nia/configuration#backend). The data persists if CTS stops and the backend is configured to a remote location. ### CTS task and event data By default, CTS stores task and event data in the Consul KV. This data is transient and does not persist unless you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha). High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. -The data stored when operating in high availability mode includes task changes made using the task API or CLI, such as creating a new task, deleting a task, or enabling/disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). +The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). ## Instance compatibility checks (high availability) -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently, enabling CTS to properly perform automations configured in the state storage. +If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage. The only incompatibility CTS checks for are tasks that are configured with a [local module](/docs/nia/configuration#module). CTS instances that do not include this module directory are incompatible. Example log: @@ -72,7 +72,7 @@ Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. -CTS will continue to run and log the error message if it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility will not run successfully. This is because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. +CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This happens because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. ## Security guidelines diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 801b14ab6f..21730c3907 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -25,7 +25,7 @@ The following table describes all of the available flags. | --- | ---- | ---- | ---- | ---- | | `-config-dir`| Required when `-config-file` is not set | string | Specifies a directory containing CTS instance configuration files to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one directory of files. | none | | `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | -| `-inspect` | Optional | boolean | Starts CTS in inspect mode (refer to [Modes](#modes) for additional information). In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | +| `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | | `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | | `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | @@ -38,6 +38,6 @@ By default, CTS starts in long-running mode. The following table describes all a | Name | Description |How to start | | --- | --- | --- | | Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | -| Once mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | +| Once-mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | | Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| | High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 7b665205d6..dbf6c4530e 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -226,22 +226,22 @@ The `cluster` parameter contains configurations for the cluster you want to oper | Parameter | Description| Required | Type | | --------- | ---------- | -------- | ------| | `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | -| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information | Optional | Object | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | -#### High availability cluster storage +#### `high_availability.cluster.storage` The `high_availability.cluster.storage` object contains the following configurations. | Parameter | Description| Required | Type | | --------- | ---------- | -------- | ------| | `parent_path` | Defines a parent path in the Consul KV for CTS to store state information. Default is `consul-terraform-sync/`. CTS automatically appends the cluster name to the parent path, so the effective default directory for state information is `consul-terraform-sync/`. | Optional | String | -| `namespace` | specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | -| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | `15s` | +| `namespace` | Specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | +| `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | String | ### `high_availbility.instance` The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: -* `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. +- `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. ## Service @@ -272,7 +272,7 @@ service { ## Task -A `task` block configures which task to execute in automation. When the task executes can be determined by the `condition` block. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). +A `task` block configures which task to execute in automation. Use the `condition` block to specify when the task executes. You can specify the `task` block multiple times to configure multiple tasks, or you can omit it entirely. If task blocks are not specified in your initial configuration, you can add them to a running CTS instance by using the [`/tasks` API endpoint](/docs/nia/api/tasks#tasks) or the [CLI's `task` command](/docs/nia/cli/task#task). ```hcl task { diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index c4aa3daa7b..e7ca2e3631 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -23,9 +23,9 @@ The resolution is to create the missing local module on the incompatible CTS ins In the following example response error: -* CTS can determine the leader. -* `high_availability.instance.address` is configured for the leader. -* The CTS instance you sent the request to is not the leader. +- CTS can determine the leader. +- `high_availability.instance.address` is configured for the leader. +- The CTS instance you sent the request to is not the leader. ```json diff --git a/website/content/docs/nia/usage/requirements.mdx b/website/content/docs/nia/usage/requirements.mdx index 46c20da43c..eeb6796649 100644 --- a/website/content/docs/nia/usage/requirements.mdx +++ b/website/content/docs/nia/usage/requirements.mdx @@ -9,13 +9,13 @@ description: >- The following components are required to run Consul-Terraform-Sync (CTS): -* A Terraform Provider -* A Terraform Module -* A Consul cluster running outside of the `consul-terraform-sync` daemon +- A Terraform provider +- A Terraform module +- A Consul cluster running outside of the `consul-terraform-sync` daemon You can add support for your network infrastructure through Terraform providers so that you can apply Terraform modules to implement network integrations. -The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driverhas [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). +The following guidance is for running CTS using the Terraform driver. The Terraform Cloud driver has [additional prerequisites](/docs/nia/network-drivers/terraform-cloud#setting-up-terraform-cloud-driver). ## Run a Consul cluster @@ -33,7 +33,7 @@ For information on compatible Consul versions, refer to the [Consul compatibilit The Consul agent must be running in order to dynamically update network devices. Refer to the [Consul agent documentation](/docs/agent/index) for information about configuring and starting a Consul agent. For hands-on instructions about running Consul agents, refer to the [Getting Started: Run the Consul Agent Tutorial](https://learn.hashicorp.com/tutorials/consul/get-started-agent?in=consul/getting-started). -When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitors a high number of services. +When running a Consul agent with CTS in production, consider that CTS uses [blocking queries](/api-docs/features/blocking) to monitor task dependencies, such as changes to registered services. This results in multiple long-running TCP connections between CTS and the agent to poll changes for each dependency. Consul may quickly reach the agent connection limits if CTS is monitoring a high number of services. To avoid reaching the limit prematurely, we recommend using HTTP/2 (requires HTTPS) to communicate between CTS and the Consul agent. When using HTTP/2, CTS establishes a single connection and reuses it for all communication. Refer to the [Consul Configuration section](/docs/nia/configuration#consul) for details. @@ -41,7 +41,7 @@ Alternatively, you can configure the [`limits.http_max_conns_per_client`](/docs/ ### Register services -CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon will be operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. +CTS monitors the Consul catalog for service changes that lead to downstream changes to your network devices. Without services, your CTS daemon is operational but idle. You can register services with your Consul agent by either loading a service definition or by sending an HTTP API request. The following HTTP API request example registers a service named `web` with your Consul agent: @@ -68,7 +68,7 @@ For hands-on instructions on registering a service by loading a service definiti For production environments, we recommend operating a Consul cluster rather than a single agent. Refer to [Getting Started: Create a Local Consul Datacenter](https://learn.hashicorp.com/tutorials/consul/get-started-create-datacenter?in=consul/getting-started) for instructions on starting multiple Consul agents and joining them into a cluster. -## Network infrastructure (using a Terraform provider) +## Network infrastructure using a Terraform provider CTS integrations for the Terraform driver use Terraform providers as plugins to interface with specific network infrastructure platforms. The Terraform driver for CTS inherits the expansive collection of Terraform providers to integrate with. You can also specify a provider `source` in the [`required_providers` configuration](https://www.terraform.io/language/providers/requirements#requiring-providers) to use providers written by the community (requires Terraform 0.13 or later). @@ -78,16 +78,16 @@ To find providers for the infrastructure platforms you use, browse the providers ### How to create a provider -If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registery so that you can use it within a network integration task or createa compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: +If a Terraform provider does not exist for your environment, you can create a new Terraform provider and publish it to the registry so that you can use it within a network integration task or create a compatible Terraform module. Refer to the following Terraform tutorial and documentation for additional information on creating and publishing providers: -* [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) -* [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). +- [Setup and Implement Read](https://learn.hashicorp.com/tutorials/terraform/provider-setup) +- [Publishing Providers](https://www.terraform.io/docs/registry/providers/publishing.html). -## Network integration (using a Terraform module) +## Network integration using a Terraform module -The Terraform module for a task in CTS is the core component of the integration. It declares which resources and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the condition under which your infrastructure is updated. +The Terraform module for a task in CTS is the core component of the integration. It declares which resources to use and how your infrastructure is dynamically updated. The module, along with how it is configured within a task, determines the conditions under which your infrastructure is updated. -Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver or use a module built by partners below. +Working with a Terraform provider, you can write an integration task for CTS by [creating a Terraform module](/docs/nia/terraform-modules) that is compatible with the Terraform driver. You can also use a [module built by partners](#partner-terraform-modules). Refer to [Configuration](/docs/nia/configuration) for information about configuring CTS and how to use Terraform providers and modules for tasks. @@ -118,7 +118,7 @@ The modules listed below are available to use and are compatible with CTS. #### Citrix ADC -- Create, Update and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) +- Create, Update, and Delete Service Groups in Citrix ADC: [Terraform Registry](https://registry.terraform.io/modules/citrix/servicegroup-consul-sync-nia/citrixadc/latest) / [GitHub](https://github.com/citrix/terraform-citrixadc-servicegroup-consul-sync-nia) #### F5 @@ -126,7 +126,7 @@ The modules listed below are available to use and are compatible with CTS. #### NS1 -- Create, Delete and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) +- Create, Delete, and Update DNS Records and Zones: [Terraform Registry](https://registry.terraform.io/modules/ns1-terraform/record-sync-nia/ns1/latest) / [GitHub](https://github.com/ns1-terraform/terraform-ns1-record-sync-nia) #### Palo Alto Networks diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index ef4b46eca0..ba8494b85c 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -11,7 +11,7 @@ This topic describes how to run Consul-Terraform-Sync (CTS) configured for high ## Introduction -A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process if it is configured for high availability: +A network always has exactly one instance of the CTS cluster that is the designated leader. The leader is responsible for monitoring and running tasks. If the leader fails, CTS triggers the following process when it is configured for high availability: 1. The CTS cluster promotes a new leader from the pool of followers in the network. 1. The new leader begins running all existing tasks in `once-mode` in order to process changes that occurred during the failover transition period. In this mode, CTS runs all existing tasks one time. @@ -29,11 +29,11 @@ The following diagram shows the CTS cluster state after the leader stops. CTS In ### Failover details -* The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. -* If failover occurs during task execution, a new leader is elected and will attempt to run all tasks once before continuing to monitor for changes. -* If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. -* If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. -* If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. +- The time it takes for a new leader to be elected is determined by the `high_availability.cluster.storage.session_ttl` configuration. The minimum failover time is equal to the `session_ttl` value. The maximum failover time is double the `session_ttl` value. +- If failover occurs during task execution, a new leader is elected. The new leader will attempt to run all tasks once before continuing to monitor for changes. +- If using the [Terraform Cloud (TFC) driver](/docs/nia/network-drivers/terraform-cloud), the task finishes and CTS starts a new leader that attempts to queue a run for each task in TFC in once-mode. +- If using [Terraform driver](/docs/nia/network-drivers/terraform), the task may complete depending on the cause of the failover. The new leader starts and attempts to run each task in [once-mode](/docs/nia/cli/start#modes). Depending on the module and provider, the task may require manual intervention to fix any inconsistencies between the infrastructure and Terraform state. +- If failover occurs when no task is executing, CTS elects a new leader that attempts to run all tasks in once-mode. Note that driver behavior is consistent whether or not CTS is running in high availability mode. @@ -46,7 +46,7 @@ Note that driver behavior is consistent whether or not CTS is running in high av You must configure appropriate ACL permissions for your cluster. Refer to [ACL permissions](#) for details. -It’s not required, but we recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. +We recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-cloud) in your CTS configuration if you want to run in high availability mode. ## Configuration @@ -97,7 +97,7 @@ We recommend deploying a cluster that includes three CTS instances. This is so t ## Modify an instance configuration -You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings, when high availability is enabled. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). +You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). 1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: ```shell-session @@ -111,7 +111,9 @@ You can implement a rolling update to update a non-task configuration for a CTS ## Modify tasks -When high availability is enabled, CTS persists task and event data (refer to State storage and persistence for additional information). You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes. This is to limit inconsistencies between the state and the configuration that can occur when mixing methods. +When high availability is enabled, CTS persists task and event data. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. + +You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes because inconsistencies between the state and the configuration can occur when mixing methods. ### Delete and recreate the task (recommended) @@ -137,19 +139,9 @@ Use the CTS API to identify the CTS leader instance and delete and replace a tas --data @payload.json \ localhost:8558/v1/tasks ``` + You can also use the [`task-create` command](/docs/nia/cli/task#task-create) to complete this step. -Send a `POST` call to the `/task/` endpoint and include the updated task in your payload. - -```shell-session -$curl --header "Content-Type: application/json" \ ---request POST \ ---data @payload.json \ - localhost:8558/v1/tasks -``` - -You can also use the `task-create` command to complete this step. - ### Discard data with the `-reset-storage` flag You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/options) to discard persisted data if you need to update a task. @@ -181,4 +173,4 @@ Use the following troubleshooting procedure if a previous leader had been runnin 1. Check for differences between the previous leader and new leader, such as differences in configurations, environment variables, and local resources. 1. Start a new instance with the fix that resolves the issue. 1. Tear down the leader instance that has the issue and any other instances that may have the same issue. -1. Restart the instance(s) to implement the fix. +1. Restart the affected instances to implement the fix. diff --git a/website/content/docs/nia/usage/run.mdx b/website/content/docs/nia/usage/run.mdx index a90cfae1af..ef26f96c56 100644 --- a/website/content/docs/nia/usage/run.mdx +++ b/website/content/docs/nia/usage/run.mdx @@ -23,7 +23,7 @@ This topic describes the basic procedure for running Consul-Terraform-Sync (CTS) $ consul-terraform-sync start -config-file ``` -4. Check status of tasks. Replace port number if configured in Step 2. See additional API endpoints [here](/docs/nia/api) +4. Check status of tasks. Replace port number if configured in Step 2. Refer to [Consul-Terraform-Sync API](/docs/nia/api) for additional information. ```shell-session $ curl localhost:8558/status/tasks From 88f007669f418bf3f4b95fc52a0d76b9e27f68ed Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 12:18:28 -0700 Subject: [PATCH 42/57] Apply suggestions from code review Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --- website/content/docs/nia/configuration.mdx | 8 ++++---- website/content/docs/nia/usage/run-ha.mdx | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index dbf6c4530e..c08479e3ff 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -7,7 +7,7 @@ description: >- # Consul-Terraform-Sync Configuration -This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL and JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. +This topic contains configuration reference information for Consul-Terraform-Sync (CTS). Pass configuration settings in an HCL or JSON configuration file to configure the CTS daemon. Refer to the [HashiCorp Configuration Language](https://github.com/hashicorp/hcl) to learn the HCL syntax. ## Global configurations @@ -219,7 +219,7 @@ The default health check is an [HTTP check](/docs/discovery/checks#http-interval Add a `high_availability` block to your configuration to enable CTS to run in high availability mode. Refer to [Run Consul-Terrform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. The `high_availability` block contains the following configuration items. -### `high_availability.cluster` +### High availability cluster The `cluster` parameter contains configurations for the cluster you want to operate with high availability enabled. You can configure the following options: @@ -238,7 +238,7 @@ The `high_availability.cluster.storage` object contains the following configurat | `namespace` | Specifies the namespace to use when storing state in the Consul KV. Default is inferred from the CTS ACL token. The fallback default is `default`. | Optional | String | | `session_ttl` | Specifies the session time-to-live for leader elections. You must specify a value greater than the `session_ttl_min` configured for Consul. A longer `session_ttl` results in a longer leader election after a failover. Default is `15s`. | Optional | String | -### `high_availbility.instance` +### High availability instance The `instance` parameter is an object that contains configurations unique to the CTS instance. You specify the following configurations: - `address`: (Optional) String value that specifies the IP address of the CTS instance to advertise to other instances. This parameter does not have a default value. @@ -655,7 +655,7 @@ driver "terraform" { - Supported backend options: [azurerm](https://www.terraform.io/docs/backends/types/azurerm.html), [consul](https://www.terraform.io/docs/backends/types/consul.html), [cos](https://www.terraform.io/docs/backends/types/cos.html), [gcs](https://www.terraform.io/docs/backends/types/gcs.html), [kubernetes](https://www.terraform.io/docs/backends/types/kubernetes.html), [local](https://www.terraform.io/docs/backends/types/local.html), [manta](https://www.terraform.io/docs/backends/types/manta.html), [pg](https://www.terraform.io/docs/backends/types/pg.html) (Terraform v0.14+), [s3](https://www.terraform.io/docs/backends/types/s3.html). Visit the Terraform documentation links for details on backend configuration options. - If omitted, CTS will generate default values and use configurations from the [`consul` block](#consul) to configure [Consul as the backend](https://www.terraform.io/docs/backends/types/consul.html), which stores Terraform statefiles in the Consul KV. The [ACL token provided for Consul authentication](#consul) is used to read and write to the KV store and requires [Consul KV privileges](https://learn.hashicorp.com/tutorials/consul/consul-terraform-sync-secure?utm_source=WEBSITE&utm_medium=WEB_IO&utm_offer=ARTICLE_PAGE&utm_content=DOCS#configure-acl-privileges-for-consul-terraform-sync). The Consul KV path is the base path to store state files for tasks. The full path of each state file will have the task identifier appended to the end of the path, e.g. `consul-terraform-sync/terraform-env:task-name`. - The remote enhanced backend is not supported with the Terraform driver to run operations in Terraform Cloud. Use the [Terraform Cloud driver](#terraform-cloud-driver) to integrate CTS with Terraform Cloud for remote workspaces and remote operations. - - The `local` backend type is not compatible with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. Refer to [Instance compatibility checks](/docs/nia/architecture#instance-compatibility-checks-high-availability) for additional information. + - The `local` backend type is not supported with CTS instances configured for high availability. If high availability is configured and the Terraform backend type is `local`, CTS logs an error and exits. - `log` - (bool) Enable all Terraform output (stderr and stdout) to be included in the CTS log. This is useful for debugging and development purposes. It may be difficult to work with log aggregators that expect uniform log format. - `path` - (string) The file path to install Terraform or discover an existing Terraform binary. If omitted, Terraform will be installed in the same directory as the CTS daemon. To resolve an incompatible Terraform version or to change versions will require removing the existing binary or change to a different path. - `persist_log` - (bool) Enable trace logging for each Terraform client to disk per task. This is equivalent to setting `TF_LOG_PATH=/terraform.log`. Trace log level results in verbose logging and may be useful for debugging and development purposes. We do not recommend enabling this for production. There is no log rotation and may quickly result in large files. diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index ba8494b85c..87c6c68447 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -50,7 +50,7 @@ We recommend specifying the [TFC driver](/docs/nia/network-drivers/terraform-clo ## Configuration -Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](#) for details about the configuration fields for the `high_availability` block. +Add the `high_availability` block in your CTS configuration and configure the required settings to enable high availability. Refer to the [Configuration reference](/docs/nia/configuration#high-availability) for details about the configuration fields for the `high_availability` block. The following example configures high availability functionality for a cluster named `cts-cluster`: @@ -117,9 +117,9 @@ You can use the following methods for modifying tasks when high availability is ### Delete and recreate the task (recommended) -Use the CTS API to identify the CTS leader instance and delete and replace a task. +Use the CTS API to identify the CTS leader instance as well as delete and replace a task. -1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: +1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= @@ -144,7 +144,7 @@ Use the CTS API to identify the CTS leader instance and delete and replace a tas ### Discard data with the `-reset-storage` flag -You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/options) to discard persisted data if you need to update a task. +You can restart the CTS cluster using the [`-reset-storage` flag](/docs/nia/cli/start#options) to discard persisted data if you need to update a task. 1. Stop a follower instance. 1. Update the instance’s task configuration. From 43302fe33325aa221271d6536566fa971ba4fb8e Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 12:20:29 -0700 Subject: [PATCH 43/57] Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> --- website/content/docs/nia/cli/start.mdx | 2 +- website/content/docs/nia/compatibility.mdx | 1 + website/content/docs/nia/usage/errors-ref.mdx | 1 - 3 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 21730c3907..9b57be03fc 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -27,7 +27,7 @@ The following table describes all of the available flags. | `-config-file` | Required when `-config-dir` is not set | string | Specifies the CTS instance configuration file to load on startup. Files must be in HCL or JSON format. You can specify the flag multiple times to load more than one file. | none | | `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | -| `-once` | Optional | boolean | Starts CTS in once-mode (refer to [Modes](#modes) for additional information). In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods. | `false` | +| `-once` | Optional | boolean | Starts CTS in once-mode. In once-mode, CTS renders templates, runs tasks once, and disables buffer periods. Refer to [Modes](#modes) for additional information. | `false` | | `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | | `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | diff --git a/website/content/docs/nia/compatibility.mdx b/website/content/docs/nia/compatibility.mdx index 4443085af6..05e301e028 100644 --- a/website/content/docs/nia/compatibility.mdx +++ b/website/content/docs/nia/compatibility.mdx @@ -26,6 +26,7 @@ CTS integration with Terraform is supported for the following: | CTS Version | Terraform CLI Version | Terraform Cloud Version | Terraform Enterprise Version | | :------------------ | :-------------------- | :---------------------- | :--------------------------- | +| CTS Enterprise 0.7 | 0.13-1.2 | Latest | v202010-2 - Latest | | CTS Enterprise 0.4+ | 0.13 - 1.1 | Latest | v202010-2 - Latest | | CTS Enterprise 0.3 | 0.13 - 1.1 | N/A | v202010-2 - Latest | | CTS OSS 0.7 | 0.13 - 1.2 | N/A | N/A | diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index e7ca2e3631..a456736c2c 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -49,7 +49,6 @@ In the following example response error: * `high_availability.instance.address` is not configured. ```json - { "error": { "message": "redirect requests to leader 'cts-01'" From 5c1f63ddfbad31164aef50e5af9b0df90385215b Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 14:28:20 -0700 Subject: [PATCH 44/57] Apply suggestions from code review Co-authored-by: lornasong --- website/content/docs/nia/architecture.mdx | 6 +++--- website/content/docs/nia/cli/start.mdx | 8 ++++---- website/content/docs/nia/configuration.mdx | 7 +++---- website/content/docs/nia/usage/errors-ref.mdx | 2 +- website/content/docs/nia/usage/run-ha.mdx | 8 +++++--- 5 files changed, 16 insertions(+), 15 deletions(-) diff --git a/website/content/docs/nia/architecture.mdx b/website/content/docs/nia/architecture.mdx index 89514908e5..39ba71f654 100644 --- a/website/content/docs/nia/architecture.mdx +++ b/website/content/docs/nia/architecture.mdx @@ -55,7 +55,7 @@ By default, CTS stores [Terraform state data](https://www.terraform.io/docs/stat ### CTS task and event data -By default, CTS stores task and event data in the Consul KV. This data is transient and does not persist unless you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha). High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. +By default, CTS stores task and event data in memory. This data is transient and does not persist. If you configure [CTS to run with high availability enabled](/docs/nia/usage/run-ha), CTS stores the data in the Consul KV. High availability is an enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops. The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the [`-reset-storage` flag](/docs/nia/cli/start#options). @@ -63,7 +63,7 @@ The data stored when operating in high availability mode includes task changes m If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage. -The only incompatibility CTS checks for are tasks that are configured with a [local module](/docs/nia/configuration#module). CTS instances that do not include this module directory are incompatible. Example log: +The CTS instance compatibility check reports an error if the task [module](/docs/nia/configuration#module) is configured with a local module, but the module does not exist on the CTS instance. Refer to the [Terraform documentation](https://www.terraform.io/language/modules/sources#module-sources) for additional information about module sources. Example log: ```shell-session [ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" @@ -72,7 +72,7 @@ Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information CTS instances perform a compatibility check on start-up based on the stored state and every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it. -CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This happens because all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. +CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This can happen when all active CTS instances enter [`once-mode`](/docs/nia/cli/start#modes) and run the tasks once when initially elected. ## Security guidelines diff --git a/website/content/docs/nia/cli/start.mdx b/website/content/docs/nia/cli/start.mdx index 9b57be03fc..c47a183df6 100644 --- a/website/content/docs/nia/cli/start.mdx +++ b/website/content/docs/nia/cli/start.mdx @@ -28,7 +28,7 @@ The following table describes all of the available flags. | `-inspect` | Optional | boolean | Starts CTS in inspect mode . In inspect mode, CTS displays the proposed state changes for all tasks once and exits. No changes are applied. Refer to [Modes](#modes) for additional information. If an error occurs before displaying all changes, CTS exits with a non-zero status. | `false` | | `-inspect-task` | Optional | string | Starts CTS in inspect mode for the specified task. CTS displays the proposed state changes for the specified task and exits. No changes are applied.
You can specify the flag multiple times to display more than one task.
If an error occurs before displaying all changes, CTS exits with a non-zero status. | none | | `-once` | Optional | boolean | Starts CTS in once-mode. In once-mode, CTS renders templates, runs tasks once, and disables buffer periods. Refer to [Modes](#modes) for additional information. | `false` | -| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | false | +| `-reset-storage` | Optional | boolean | Directs CTS to overwrite the state storage with new state information when the instance you are starting is elected the cluster leader.
Only use this flag when running CTS in [high availability mode](/docs/nia/usage/run-ha). | `false` | | `-h`, `-help` | Optional | boolean | Prints the CTS command line help. | `false` | ## Modes @@ -37,7 +37,7 @@ By default, CTS starts in long-running mode. The following table describes all a | Name | Description |How to start | | --- | --- | --- | -| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process in which it logs errors and exits.

When the long-running process begins, the CTS daemon serves API and command requests.

| No additional flags.
This is the default mode. | +| Long-running mode | CTS starts in once-mode and switches to a long-running process.

During the once-mode phase, the daemon exits with a non-zero status if it encounters an error.

After successfully operating in once-mode, CTS begins a long-running process.

When the long-running process begins, the CTS daemon serves API and command requests.

If an error occurs, CTS logs it and continues running.

| No additional flags.
This is the default mode. | | Once-mode | In once-mode, CTS renders templates and runs tasks once. CTS does not start the process in long-running mode and disables buffer periods.

Use once-mode before starting CTS in long-running mode to verify that your configuration is accurate and tasks update network infrastructure as expected.

| Add the `-once` flag when starting CTS. | -| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| -| High availability mode | Ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| +| Inspect mode | CTS displays the proposed state changes for all tasks once and exits. No changes are applied. If an error occurs before displaying all changes, CTS exits with a non-zero status.

Use inspect mode before starting CTS in long-running mode to debug one or more tasks and to verify that your tasks will update network infrastructure as expected.

| Add the `-inspect` flag to verify all tasks.

Add the `-inspect-task` flag to inspect a single task. Use multiple flags to verify more than one task.

| +| High availability mode | A long-running process that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. CTS logs the errors and continues to operate without interruption. Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information. | Add the `high_availability` block to your CTS instance configuration.

Refer to [Run Consul-Terraform-Sync with High Availability](/docs/nia/usage/run-ha) for additional information.

| diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index c08479e3ff..90e5905153 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -44,7 +44,7 @@ tls { - `max` - (string: "20s") The maximum period of time to wait after changes are detected before triggering related tasks. If `min` is set, the default period for `max` is 4 times the value of `min`. - `log_level` - (string: "INFO") The log level to use for CTS logging. This defaults to "INFO". The available log levels are "TRACE", "DEBUG", "INFO", "WARN", and "ERR". - `port` - (int: 8558) The port for CTS to use to serve API requests. -- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in the cluster. +- `id` (string: Generated ID with the format `cts-`) The ID of the CTS instance. CTS uses the ID as the service ID for CTS if service registration is enabled. CTS also uses the ID to identify the instance in a high availability cluster. - `syslog` - Specifies the syslog server for logging. - `enabled` - (bool) Enable syslog logging. Specifying other option also enables syslog logging. - `facility` - (string: "local0") Name of the syslog facility to log to. @@ -190,8 +190,7 @@ Service registration requires that the [Consul token](/docs/nia/configuration#co | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | -| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. -| `true` | +| `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. | `true` | | `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high_availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | @@ -226,7 +225,7 @@ The `cluster` parameter contains configurations for the cluster you want to oper | Parameter | Description| Required | Type | | --------- | ---------- | -------- | ------| | `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | -| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `”consul”` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | +| `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `"consul"` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | #### `high_availability.cluster.storage` diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index a456736c2c..2dc82d7925 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -17,7 +17,7 @@ If you configured the CTS cluster to run in [high availability mode](/docs/nia/u [ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory" ``` -The resolution is to create the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. +The resolution is to add the missing local module on the incompatible CTS instance. Refer to the [`module` documentation](/docs/nia/configuration#module) in the CTS configuration reference for additional information. ## Example API and CLI error messages diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 87c6c68447..52c2c77268 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -6,7 +6,9 @@ description: >- --- # Run Consul-Terraform-Sync with High Availability - + + Licenses are only required for Consul-Terraform-Sync (CTS) Enterprise + This topic describes how to run Consul-Terraform-Sync (CTS) configured for high availability. High availability is an enterprise capability that ensures that all changes to Consul that occur during a failover transition are processed and that CTS continues to operate as expected. ## Introduction @@ -19,7 +21,7 @@ A network always has exactly one instance of the CTS cluster that is the designa In a standard configuration, CTS exits if errors occur when the CTS instance runs tasks in `once-mode`. In a high availability configuration, CTS logs the errors and continues to operate without interruption. -The following diagram shows operating state when high availability is enabled: +The following diagram shows operating state when high availability is enabled. CTS Instance A is the current leader and is responsible for monitoring and running tasks: ![Consul-Terraform-Sync architecture configured for high availability before a shutdown event](/img/nia/cts-ha-before.svg) @@ -89,7 +91,7 @@ We recommend deploying a cluster that includes three CTS instances. This is so t ```shell-session $ consul-terraform-sync start -config-file ha-config.hcl ``` -1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. +1. You can call the `/status` API endpoint to verify the status of tasks CTS is configured to monitor. Only the leader of the cluster will return a successful response. Refer to the [`/status` API reference documentation](/docs/nia/api/status) for information about usage and responses. ```shell-session $ curl localhost:/status/tasks From a7a0acf754bc019402f59b67aa77acb01e78823b Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Fri, 26 Aug 2022 14:46:13 -0700 Subject: [PATCH 45/57] final content tweaks for CTS 0.7 docs --- website/content/docs/nia/usage/errors-ref.mdx | 55 ++++++++++++------- website/content/docs/nia/usage/run-ha.mdx | 5 +- 2 files changed, 40 insertions(+), 20 deletions(-) diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 2dc82d7925..9c7cd8bf89 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -21,12 +21,7 @@ The resolution is to add the missing local module on the incompatible CTS instan ## Example API and CLI error messages -In the following example response error: - -- CTS can determine the leader. -- `high_availability.instance.address` is configured for the leader. -- The CTS instance you sent the request to is not the leader. - +**Error**: ```json { @@ -36,17 +31,25 @@ In the following example response error: } ``` -The resolution is to redirect the request to the leader instance, for example: +**Conditions**: + +- CTS can determine the leader. +- `high_availability.instance.address` is configured for the leader. +- The CTS instance you sent the request to is not the leader. + + + +**Resolution**: + +Redirect the request to the leader instance, for example: ```shell-session $ curl --request GET cts-01.example.com:8558/v1/tasks ``` --- -In the following example response error: -* CTS can determine the leader. -* The CTS instance you sent the request to is not the leader. -* `high_availability.instance.address` is not configured. + +**Error**: ```json { @@ -55,15 +58,20 @@ In the following example response error: } } ``` -The resolution is to identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. + +**Conditions**: + +* CTS can determine the leader. +* The CTS instance you sent the request to is not the leader. +* `high_availability.instance.address` is not configured. + +**Resolution**: + +Identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. --- -In the following example response error: - -* The CTS instance you sent the request to is not the leader. -* The CTS is unable to determine the leader. -* Note that these conditions are rare. +**Error**: ```json { @@ -72,9 +80,18 @@ In the following example response error: } } ``` -The resolution is to identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: + +**Conditions**: + +* The CTS instance you sent the request to is not the leader. +* The CTS is unable to determine the leader. +* Note that these conditions are rare. + + +**Resolution**: + +Identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= ``` - diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 52c2c77268..23ba1d414f 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -41,7 +41,8 @@ Note that driver behavior is consistent whether or not CTS is running in high av ## Requirements -* Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. +Verify that you have met the [basic requirements](/docs/nia/usage/requirements) for running CTS. + * CTS Enterprise 0.7 or later * Terraform CLI 0.13 or later * All instances in a cluster must be in the same datacenter. @@ -97,6 +98,8 @@ We recommend deploying a cluster that includes three CTS instances. This is so t $ curl localhost:/status/tasks ``` +Repeat the procedure to start the remaining instances for your cluster. + ## Modify an instance configuration You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). From b153c965cbaf0fad178ee6f2e3e0242a51342f7f Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Fri, 26 Aug 2022 14:53:12 -0700 Subject: [PATCH 46/57] redirects for CTS 0.7 --- website/redirects.js | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/website/redirects.js b/website/redirects.js index c9c5b668b0..e480640b67 100644 --- a/website/redirects.js +++ b/website/redirects.js @@ -1532,6 +1532,13 @@ module.exports = [ { source: '/docs/k8s/installation/vault/wan-federation', destination: '/docs/k8s/deployment-configurations/vault/wan-federation', + source: '/docs/nia/installation/run', + destination: '/docs/nia/usage/run', + permanent: true, + }, + { + source: '/docs/nia/installation/requirements', + destination: '/docs/nia/usage/requirements', permanent: true, }, ] From d4445f10299ce4d5950256d888260db1f9fff86a Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Fri, 26 Aug 2022 15:48:55 -0700 Subject: [PATCH 47/57] Update website/content/docs/nia/usage/run-ha.mdx --- website/content/docs/nia/usage/run-ha.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index 23ba1d414f..e7dac83c0a 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -98,7 +98,7 @@ We recommend deploying a cluster that includes three CTS instances. This is so t $ curl localhost:/status/tasks ``` -Repeat the procedure to start the remaining instances for your cluster. +Repeat the procedure to start the remaining instances for your cluster. We recommend using near-identical configurations for all instances in your cluster. You may not be able to use exact configurations in all cases, but starting instances with the same configuration improves consistency and reduces confusion if you need to troubleshoot errors. ## Modify an instance configuration From 5db024004945d4561a0ec234db1564aaf7a874d8 Mon Sep 17 00:00:00 2001 From: Tu Nguyen Date: Mon, 29 Aug 2022 15:33:22 -0700 Subject: [PATCH 48/57] Apply suggestions from code review --- website/content/docs/nia/api/status.mdx | 2 +- website/content/docs/nia/configuration.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index d364f93bec..c66a63735a 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -6,7 +6,7 @@ description: >- --- # Status -The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, see [Status Information](/docs/nia/tasks#status-information). +The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 90e5905153..3682d91fdd 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -245,7 +245,7 @@ The `instance` parameter is an object that contains configurations unique to the ## Service -~> Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. See [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. +~> **Note:** Deprecated in CTS 0.5.0 and will be removed in a future major release. `service` blocks are used to define the `task` block's `services` fields, which were also deprecated and replaced with [Services Condition](/docs/nia/configuration#services-condition) and [Services Module Input](/docs/nia/configuration#services-module-input). `service` block configuration can be replaced by configuring the equivalent fields of the corresponding Services Condition and Services Module Input. Refer to [0.5.0 release notes](/docs/nia/release-notes/0-5-0#deprecate-service-block) for examples. A `service` block is an optional block to explicitly define the services configured in the `task` block's `services` field (deprecated). `service` blocks do not define services configured in the `task` block's `condition "services"` or `module_input "services` blocks. From 3935eca83aee8ffec15dd29df0114eed760c91f0 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 11:55:07 -0700 Subject: [PATCH 49/57] added cluster endpoint to status API docs --- website/content/docs/nia/api/status.mdx | 92 ++++++++++++++++++++++--- 1 file changed, 81 insertions(+), 11 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index c66a63735a..12ca873467 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -10,24 +10,24 @@ The `/status` endpoints return status-related information for tasks. This endpoi If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. -## Overall Status +## Status for all tasks -This endpoint currently returns the overall status information for all tasks. +This endpoint returns the overall status information for all tasks. | Method | Path | Produces | | ------ | ------------------- | ------------------ | | `GET` | `/status` | `application/json` | -### Request Parameters +### Request parameters Currently no request parameters are offered for the overall status API. -### Response Statuses +### Response statuses | Status | Reason | | ------ | ---------------- | | 200 | Successfully retrieved the status | -### Response Fields +### Response fields | Name | Type | Description | | ----- | ------ | ------------------ | @@ -65,7 +65,7 @@ Response: } ``` -## Task Status +## Task status This endpoint returns the individual task status information for a single specified task or for all tasks. @@ -79,7 +79,7 @@ Task health status value is determined by the success or failure of all stored [ | ------ | ------------------- | ------------------ | | `GET` | `/status/tasks/:task_name` | `application/json` | -### Request Parameters +### Request parameters | Name | Required | Type | Description | Default | | -------- | -------- | ------ | ------------------ | ------- | @@ -87,14 +87,14 @@ Task health status value is determined by the success or failure of all stored [ |`include` | Optional | string | Only accepts the value `"events"`. Use to include stored event information in response. | none |`status` | Optional | string | Only accepts health status values `"successful"`, `"errored"`, `"critical"`, or `"unknown"`. Use to filter response by tasks that have the specified health status value. Recommend setting this parameter when requesting all tasks i.e. no `task` parameter is set. | none -### Response Statuses +### Response statuses | Status | Reason | | ------ | ---------------- | | 200 | Successfully retrieved the task status | | 404 | Task with the given name not found | -### Response Fields +### Response fields The response is a JSON map of task name to a status information structure with the following fields. @@ -126,7 +126,11 @@ Event represents the process of updating network infrastructure of a task. The d |`config.source` | string | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** Module configured for the task. |`config.providers` | list[string] | **Deprecated in CTS 0.5.0 and will be removed in v0.8.0.** List of the providers configured for the task. -### Example: All Task Statuses +### Examples + +The following examples call `status` endpoints to retrieve information about tasks. + +#### All task statuses Request: ```shell-session @@ -163,7 +167,7 @@ Response: } ``` -### Example: Individual Task Status with Events +#### Individual task status with events Request: ```shell-session @@ -225,3 +229,69 @@ Response: } } ``` + +## Cluster status + +The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. + +| Method | Path | Response format | +| ------ | ----------------- | ------------------ | +| `GET` | `/status/cluster` | `application/json` | + +### Request parameters + +Currently no request parameters are offered for the overall status API. + +### Request statuses + +- `200`: Successfully retrieved the cluster status + +### Response fields + +| Field | Type | Description | +| --- | ---- | --- | +| `cluster_name` | string | Identifies the name of the cluster. | +| `members` | list | Contains the CTS instance objects. | +| `members.address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | +| `members.healthy` | Boolean | Indicates the health of the instance. | +| `members.id` | string | Indicates the instance ID. | +| `members.leader` | Boolean | Identifies the cluster leader. | +| `members.service_name` | string | Identifies the name of the service that the instance represents. The value should always be `consul-terraform-sync`. | +| `request_id` | string | Unique identifier for the request. | + +### Example + +The following command calls the `/status/cluster` endpoint: + +```shell-session +$ curl –request GET http://localhost:8553/v1/status/cluster +``` + +The following example response shows that there are three CTS instances. The cluster is located at `cts-02.example.com`. The leader instance, `cts-02`, is the only healthy instance: + +```json +{ + "cluster_name": "cluster-a", + "members": [ + { + "address": "cts-02.example.com", + "healthy": true, + "id": "cts-02", + "leader": true, + "service_name": "consul-terraform-sync" + }, + { + "healthy": false, + "id": "cts-03", + "leader": false, + "service_name": "consul-terraform-sync" + }, + { + "healthy": false, + "id": "cts-01", + "leader": false, + "service_name": "consul-terraform-sync" + } + ], + "request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88" +} From acc184bd09323ecafd988382b3c212e38b9290fc Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 11:59:21 -0700 Subject: [PATCH 50/57] added no-HA API status error message to EMs page --- website/content/docs/nia/api/status.mdx | 2 +- website/content/docs/nia/usage/errors-ref.mdx | 20 +++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index 12ca873467..0f5d737619 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -232,7 +232,7 @@ Response: ## Cluster status -The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. +The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. | Method | Path | Response format | | ------ | ----------------- | ------------------ | diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 9c7cd8bf89..4f433c9ef8 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -95,3 +95,23 @@ Identify and send the request to the leader CTS instance. You can identify the l ```shell-session [INFO] ha: acquired leadership lock: id= ``` + +--- + +**Error**: + +```json +{ + "error": { + "message": "this endpoint is only available with high availability configured" + } +} +``` + +**Conditions**: + +- CTS is not configured for [high availability](/docs/nia/usage/run-ha). + +**Resolution**: + +Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). \ No newline at end of file From 9a856781ad43f4b6c6351e95aedab4e3fae47a01 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 12:09:03 -0700 Subject: [PATCH 51/57] udpated how to ID the leader in the usage docs --- website/content/docs/nia/usage/errors-ref.mdx | 11 ++++++++--- website/content/docs/nia/usage/run-ha.mdx | 6 +++--- 2 files changed, 11 insertions(+), 6 deletions(-) diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 4f433c9ef8..01aa10b6fd 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -67,7 +67,12 @@ $ curl --request GET cts-01.example.com:8558/v1/tasks **Resolution**: -Identify the leader instance address and redirect the request to the leader. We recommend deploying a cluster that has three instances. You can query each instance until the leader responds. +Identify the leader instance address and redirect the request to the leader. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: + +```shell-session +[INFO] ha: acquired leadership lock: id=. + +We recommend deploying a cluster that has three instances. --- @@ -90,7 +95,7 @@ Identify the leader instance address and redirect the request to the leader. We **Resolution**: -Identify and send the request to the leader CTS instance. You can identify the leader by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: +Identify and send the request to the leader CTS instance. You can identify the leader by calling the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= @@ -110,7 +115,7 @@ Identify and send the request to the leader CTS instance. You can identify the l **Conditions**: -- CTS is not configured for [high availability](/docs/nia/usage/run-ha). +- You called the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) without configuring CTS for [high availability](/docs/nia/usage/run-ha). **Resolution**: diff --git a/website/content/docs/nia/usage/run-ha.mdx b/website/content/docs/nia/usage/run-ha.mdx index e7dac83c0a..81fa5c2d16 100644 --- a/website/content/docs/nia/usage/run-ha.mdx +++ b/website/content/docs/nia/usage/run-ha.mdx @@ -104,7 +104,7 @@ Repeat the procedure to start the remaining instances for your cluster. We recom You can implement a rolling update to update a non-task configuration for a CTS instance, such as the Consul connection settings. If you need to update a task in the instance configuration, refer to [Modify tasks](#modify-tasks). -1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/cli/start) or by checking the logs for the following entry: +1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= ``` @@ -121,10 +121,10 @@ When high availability is enabled, CTS persists task and event data. Refer to [S You can use the following methods for modifying tasks when high availability is enabled. We recommend choosing a single method to make all task configuration changes because inconsistencies between the state and the configuration can occur when mixing methods. ### Delete and recreate the task (recommended) - + Use the CTS API to identify the CTS leader instance as well as delete and replace a task. -1. Identify the leader CTS instance by either making a call to the [`status` API endpoint](/docs/nia/api/status) or by checking the logs for the following entry: +1. Identify the leader CTS instance by either making a call to the [`status/cluster` API endpoint](/docs/nia/api/status#cluster-status) or by checking the logs for the following entry: ```shell-session [INFO] ha: acquired leadership lock: id= From da77e68b271dd894c84ab885b53ff8b80101dcd2 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 16:00:09 -0700 Subject: [PATCH 52/57] applied AJs feedback and made a few additional improvements --- website/content/docs/nia/api/index.mdx | 27 +++++------ website/content/docs/nia/api/status.mdx | 46 +++++++++++-------- website/content/docs/nia/usage/errors-ref.mdx | 23 +++++++++- 3 files changed, 60 insertions(+), 36 deletions(-) diff --git a/website/content/docs/nia/api/index.mdx b/website/content/docs/nia/api/index.mdx index 3e38803027..07b1d396e6 100644 --- a/website/content/docs/nia/api/index.mdx +++ b/website/content/docs/nia/api/index.mdx @@ -2,32 +2,29 @@ layout: docs page_title: Consul-Terraform-Sync API description: >- - How to use the Consul-Terraform-Sync API + Consul-Terraform-Sync provides an API interface that lets you query CTS instance health, check CTS instance status, and run CTS tasks. --- -# Consul-Terraform-Sync API +# Consul-Terraform-Sync API Overview -When running in [daemon mode](/docs/nia/cli#daemon-mode), Consul-Terraform-Sync (CTS) serves an HTTP API interface. Details of the available API endpoints is available in the navigation to the left. +Consul-Terraform-Sync (CTS) provides an HTTP API interface for querying CTS instances and running and managing tasks. -### Port +## Port The API is served at the default port `8558` or a different port if set with [`port` configuration](/docs/nia/configuration#port) -### Version Prefix +## Version prefix All API routes are prefixed with `/v1/`. This documentation is for v1 of the API, which is the only version currently. Example: `localhost:8558/v1/status` -### Error +## Request ID -Successful API requests will receive a 2XX success status code. For other unsuccessful status codes, when possible, more details will be provided in a response body containing an error object. +Each call to a CTS API endpoint returns a `request_id` field. The field is a string generated by the API. Example: -Example: Status 400 Bad Request -```json -{ - "error": { - "message": "example error message: unsupported status parameter value" - } -} -``` +`"request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88"` + +## Error messages + +The API sends a response code in the 200 range if the call is successful. If the call is unsuccessful, the API sends an error message that includes additional information when possible. Refer to [Error Messages](/docs/nia/usage/errors-ref) for additional information. \ No newline at end of file diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index 0f5d737619..26459692d3 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -8,7 +8,7 @@ description: >- The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). -If you [run CTS with high availability enabled](/docs/nia/usage/run-ha), you can send requests to the `/status` endpoint on CTS leader or follower instances. Requests to a follower instance, however, return a 400 Bad Request and error message. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send a request to the [`/status/cluster` endpoint](#cluster-status) of any instance to get information about the cluster. Requests to the `status` endpoint return a 400 error, however, if high availability is enabled and the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Status for all tasks @@ -19,13 +19,12 @@ This endpoint returns the overall status information for all tasks. | `GET` | `/status` | `application/json` | ### Request parameters -Currently no request parameters are offered for the overall status API. + +No request parameters are available for this endpoint. ### Response statuses -| Status | Reason | -| ------ | ---------------- | -| 200 | Successfully retrieved the status | +- `200`: Successfully retrieved the status. ### Response fields @@ -89,10 +88,8 @@ Task health status value is determined by the success or failure of all stored [ ### Response statuses -| Status | Reason | -| ------ | ---------------- | -| 200 | Successfully retrieved the task status | -| 404 | Task with the given name not found | +- `200`: Successfully retrieved the task status +- `404`: Task with the given name not found ### Response fields @@ -232,7 +229,7 @@ Response: ## Cluster status -The `/v1/status/cluster` API endpoint returns information about the cluster and member instances, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. +The `/v1/status/cluster` API endpoint returns information about high-availability clusters and its members, such as health status and leadership. This endpoint is only supported when using CTS in [high availability mode](/docs/nia/usage/run-ha). If you call the endpoint without configuring CTS for high availability, then CTS prints an error to the console. Refer to [Error Messages](/docs/nia/usage/errors-ref) for information about CTS error messages. | Method | Path | Response format | | ------ | ----------------- | ------------------ | @@ -248,16 +245,24 @@ Currently no request parameters are offered for the overall status API. ### Response fields +The following table describes the responses that the `status/cluster` endpoint can send. + | Field | Type | Description | | --- | ---- | --- | | `cluster_name` | string | Identifies the name of the cluster. | -| `members` | list | Contains the CTS instance objects. | -| `members.address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | -| `members.healthy` | Boolean | Indicates the health of the instance. | -| `members.id` | string | Indicates the instance ID. | -| `members.leader` | Boolean | Identifies the cluster leader. | -| `members.service_name` | string | Identifies the name of the service that the instance represents. The value should always be `consul-terraform-sync`. | -| `request_id` | string | Unique identifier for the request. | +| `members` | array | Contains an array of [member objects](#member-objects). Each object in the `members` array represents a CTS instance. | + +#### Member objects + +The following table describes the fields available for objects in the `members` array. + +| Field | Type | Description | +| --- | ---- | --- | +| `address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured on the leader instance. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | +| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | +| `id` | string | Indicates the service registration ID. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | +| `leader` | boolean | Identifies the cluster leader. | +| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | ### Example @@ -267,7 +272,7 @@ The following command calls the `/status/cluster` endpoint: $ curl –request GET http://localhost:8553/v1/status/cluster ``` -The following example response shows that there are three CTS instances. The cluster is located at `cts-02.example.com`. The leader instance, `cts-02`, is the only healthy instance: +The following example response shows three CTS instances. The cluster leader is `cts-02` and is advertising that its address is `cts-02.example.com`: ```json { @@ -281,13 +286,13 @@ The following example response shows that there are three CTS instances. The clu "service_name": "consul-terraform-sync" }, { - "healthy": false, + "healthy": true, "id": "cts-03", "leader": false, "service_name": "consul-terraform-sync" }, { - "healthy": false, + "healthy": true, "id": "cts-01", "leader": false, "service_name": "consul-terraform-sync" @@ -295,3 +300,4 @@ The following example response shows that there are three CTS instances. The clu ], "request_id": "e1bc2236-3d0e-5f5e-dc51-164a1cf6da88" } +``` \ No newline at end of file diff --git a/website/content/docs/nia/usage/errors-ref.mdx b/website/content/docs/nia/usage/errors-ref.mdx index 01aa10b6fd..0b24a1189a 100644 --- a/website/content/docs/nia/usage/errors-ref.mdx +++ b/website/content/docs/nia/usage/errors-ref.mdx @@ -119,4 +119,25 @@ Identify and send the request to the leader CTS instance. You can identify the l **Resolution**: -Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). \ No newline at end of file +Configure CTS to run in [high availability mode](/docs/nia/usage/run-ha). + +--- + +**Error**: + +```json +{ + "error": { + "message": "example error message: unsupported status parameter value" + } +} +``` + +**Conditions**: + +- You sent a request to the `status` API endpoint. +- The request included an unsupported parameter value. + +**Resolution**: + +Send a new request and verify that all of the parameter values are correct. \ No newline at end of file From 3ce564354e92e625b008e7e8697c94692ff8990c Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Tue, 6 Sep 2022 16:13:28 -0700 Subject: [PATCH 53/57] fixed bad links in status/cluster endpoint ref --- website/content/docs/nia/api/status.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index 26459692d3..ad551e9ff9 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -259,10 +259,10 @@ The following table describes the fields available for objects in the `members` | Field | Type | Description | | --- | ---- | --- | | `address` | string | Indicates the location of the instance. The address is only included in the response if the `high_availability.instance.address` option is configured on the leader instance. Refer to the [high availability instance configuration](/docs/nia/configuration#high-availability-instance) reference for additional information. | -| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | -| `id` | string | Indicates the service registration ID. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. | +| `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | +| `id` | string | Indicates the service registration ID. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | | `leader` | boolean | Identifies the cluster leader. | -| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](#/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | +| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | ### Example From 3ec0f2cd4c22228b8e6a6aaf66dd699b00fc5ea6 Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Wed, 7 Sep 2022 12:53:58 -0700 Subject: [PATCH 54/57] Apply suggestions from code review Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> --- website/content/docs/nia/api/status.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/content/docs/nia/api/status.mdx b/website/content/docs/nia/api/status.mdx index ad551e9ff9..108f2c9aa0 100644 --- a/website/content/docs/nia/api/status.mdx +++ b/website/content/docs/nia/api/status.mdx @@ -8,7 +8,7 @@ description: >- The `/status` endpoints return status-related information for tasks. This endpoint returns a count of successful and failed task events that are recorded whenever tasks execute an automation. Currently, only the five most recent events are stored in Consul-Terraform-Sync (CTS). For more information on the hierarchy of status information and how it is collected, refer to [Status Information](/docs/nia/tasks#status-information). -If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send a request to the [`/status/cluster` endpoint](#cluster-status) of any instance to get information about the cluster. Requests to the `status` endpoint return a 400 error, however, if high availability is enabled and the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. +If CTS is configured [for high availability](/docs/nia/usage/run-ha), you can send requests to the [`/status/cluster` endpoint path](#cluster-status) on any CTS cluster member instance to receive information about the entire cluster. Calling the `status` endpoint path (without `/cluster`), however, returns a 400 error if the request is sent to a follower instance. The error message depends on what information the follower instance is able to obtain about the leader. Refer to [Error Messages](/docs/nia/usage/errors-ref) for more information. ## Status for all tasks @@ -237,7 +237,7 @@ The `/v1/status/cluster` API endpoint returns information about high-availabilit ### Request parameters -Currently no request parameters are offered for the overall status API. +Currently no request parameters are offered for the cluster status API. ### Request statuses @@ -250,7 +250,7 @@ The following table describes the responses that the `status/cluster` endpoint c | Field | Type | Description | | --- | ---- | --- | | `cluster_name` | string | Identifies the name of the cluster. | -| `members` | array | Contains an array of [member objects](#member-objects). Each object in the `members` array represents a CTS instance. | +| `members` | list[[member](#member-objects)] | Contains a list of [member objects](#member-objects). Each object in the `members` list represents a CTS instance. | #### Member objects @@ -262,7 +262,7 @@ The following table describes the fields available for objects in the `members` | `healthy` | boolean | Indicates the health of the service instance health. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | | `id` | string | Indicates the service registration ID. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. | | `leader` | boolean | Identifies the cluster leader. | -| `service_name` | string | Identifies the name of the service that the instance represents. Refer to [Service Registration](/docs/nia/configuration#service-registration) for additional information. Default is `consul-terraform-sync`. | +| `service_name` | string | Identifies the name of the service that the instance represents. The value is set by the `service_name` field in the [Service Registration](/docs/nia/configuration#service-registration) configuration. | ### Example From 83ccbf1635d7631a6d94f2a6a3dc0296b3f4117d Mon Sep 17 00:00:00 2001 From: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com> Date: Wed, 7 Sep 2022 14:00:56 -0700 Subject: [PATCH 55/57] Apply suggestions from code review Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com> --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index 3682d91fdd..d32a1fc97c 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -191,7 +191,7 @@ Service registration requires that the [Consul token](/docs/nia/configuration#co | Parameter | Required | Type | Description | Default | | --------- | -------- | ---- | ----------- | ------- | | `enabled` | Optional | boolean | Enables CTS to register itself as a service with Consul. When service registration is enabled for a [CTS instance configured for high availability](/docs/nia/usage/run-ha), the instance also registers itself with a new tag using the `cts-cluster:` format. | `true` | -| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high_availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | +| `service_name` | Optional | string | The service name for CTS. We recommended specifying the same name used for [`high_availability.cluster.name`](#high-availability-cluster) value if [CTS is configured for high availability](/docs/nia/usage/run-ha). | `consul-terraform-sync` | | `address` | Optional | string | The IP address or hostname for CTS. | IP address of the Consul agent node | | `namespace` | Optional | string | The namespace to register CTS in. | In order of precedence:
1. Inferred from the CTS ACL token
2. The `default` namespace. | | `default_check.enabled` | Optional | boolean | Enables CTS to create the default health check. | `true` | From 6b37a254788ed14ec3d13df963bb51f7b5d7eb09 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Wed, 7 Sep 2022 14:13:39 -0700 Subject: [PATCH 56/57] updated HA configuration ref --- website/content/docs/nia/configuration.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/content/docs/nia/configuration.mdx b/website/content/docs/nia/configuration.mdx index d32a1fc97c..36bff7f2dc 100644 --- a/website/content/docs/nia/configuration.mdx +++ b/website/content/docs/nia/configuration.mdx @@ -227,7 +227,7 @@ The `cluster` parameter contains configurations for the cluster you want to oper | `name` | Specifies the name of the cluster operating with high availability enabled. | Required | String | | `storage` | Configures how CTS stores state information. Refer to [State storage and persistence](/docs/nia/architecture#state-storage-and-persistence) for additional information. You can define storage for the `"consul"` resource. Refer to [High availability cluster storage](#high-availability-cluster-storage) for additional information. | Optional | Object | -#### `high_availability.cluster.storage` +#### High availability cluster storage The `high_availability.cluster.storage` object contains the following configurations. From 001b1ed8cf83ae54f82dfe1b5af4f8091dcb5955 Mon Sep 17 00:00:00 2001 From: trujillo-adam Date: Thu, 8 Sep 2022 11:31:08 -0700 Subject: [PATCH 57/57] fix to redirects file --- website/redirects.js | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/redirects.js b/website/redirects.js index 93ea6aba1b..c81aacbb52 100644 --- a/website/redirects.js +++ b/website/redirects.js @@ -1542,6 +1542,9 @@ module.exports = [ { source: '/docs/k8s/installation/vault/wan-federation', destination: '/docs/k8s/deployment-configurations/vault/wan-federation', + permanent: true, + }, + { source: '/docs/nia/installation/run', destination: '/docs/nia/usage/run', permanent: true,