consul/website/source/docs/platform/k8s/helm.html.md

layout	page_title	sidebar_current	description
docs	Helm - Kubernetes	docs-platform-k8s-helm	The Consul Helm chart is the recommended way to install and configure Consul on Kubernetes. In addition to running Consul itself, the Helm chart is the primary method for installing and configuring Consul integrations with Kubernetes such as catalog syncing, Connect injection, and more.

Helm Chart

The Consul Helm chart is the recommended way to install and configure Consul on Kubernetes. In addition to running Consul itself, the Helm chart is the primary method for installing and configuring Consul integrations with Kubernetes such as catalog syncing, Connect injection, and more.

This page assumes general knowledge of Helm and how to use it. Using Helm to install Consul will require that Helm is properly installed and configured with your Kubernetes cluster.

-> Important: The Helm chart is new and may still change significantly over time. Please always run Helm with --dry-run before any install or upgrade to verify changes.

~> Security Warning: By default, the chart will install an insecure configuration of Consul. This provides a less complicated out-of-box experience for new users, but is not appropriate for a production setup. Make sure that your Kubernetes cluster is properly secured to prevent unwanted access to Consul, or that you understand and enable the recommended Consul security features. Currently, some of these features are not supported in the Helm chart and require additional manual configuration.

Using the Helm Chart

To install Consul using the Helm chart you must first install Helm onto your Kubernetes cluster. See the Helm Install Guide for more information.

Once Helm is installed, determine the latest version of the Consul Helm chart by visiting https://github.com/hashicorp/consul-helm/releases.

Clone the chart at that version, for example if the latest version is v0.8.1 you would run:

$ git clone --single-branch --branch v0.8.1 https://github.com/hashicorp/consul-helm.git
Cloning into 'consul-helm'...
...
You are in 'detached HEAD' state...

Ensure you've checked out the correct version with helm inspect:

$ helm inspect chart ./consul-helm
apiVersion: v1
description: Install and configure Consul on Kubernetes.
home: https://www.consul.io
name: consul
sources:
- https://github.com/hashicorp/consul
- https://github.com/hashicorp/consul-helm
- https://github.com/hashicorp/consul-k8s
version: 0.8.1

Now you're ready to install Consul! To install Consul with the default configuration run:

$ helm install --name consul ./consul-helm
NAME:   consul
...
Your release is named consul. To learn more about the release, try:

  $ helm status consul
  $ helm get consul

If you want to customize the installation, create a values.yaml file to override the default settings. You can learn what settings are available by running helm inspect values ./consul-helm or by reading the below Configuration section.

Once you've created your values.yaml file, run helm install with the -f flag:

$ helm install --name consul -f ./values.yaml ./consul-helm

~> Warning: By default, the chart will install everything: a Consul server cluster, client agents on all nodes, feature components, etc. This provides a nice out-of-box experience for new users, but may not be appropriate for a production setup. Consider setting the global.enabled value to false and opt-in to the various components.

Configuration (Values)

The chart is highly customizable using Helm configuration values. Each value has a sane default tuned for an optimal getting started experience with Consul. Before going into production, please review the parameters below and consider if they're appropriate for your deployment.

• global - These global values affect multiple components of the chart.

  • enabled (boolean: true) - The master enabled/disabled configuration. If this is true, most components will be installed by default. If this is false, no components will be installed by default and manually opt-in is required, such as by setting server.enabled to true.

  • domain (string: "consul") - The domain Consul uses for DNS queries. This is used to configure agents both for DNS listening but also to know what domain to join the cluster. This should be consistent throughout the chart, but can be overridden per-component as well.

  • image (string: "consul:latest") - The name of the Docker image (including any tag) for the containers running Consul agents. This should be pinned to a specific version when running in production. Otherwise, other changes to the chart may inadvertently upgrade your Consul version.

  • imageK8S (string: "hashicorp/consul-k8s:latest") - The name of the Docker image (including any tag) for the consul-k8s binary. This is used by components such as catalog sync. This should be pinned to a specific version when running in production. Otherwise, other changes to the chart may inadvertently upgrade the version.

  • datacenter (string: "dc1") - The name of the datacenter that the agent cluster should register as. This may not be changed once the cluster is bootstrapped and running, since Consul doesn't yet support an automatic way to change this value.

• enablePodSecurityPolicies (boolean: false) - This flag controls whether PodSecurityPolicies are created for the Consul components that this chart creates.

• bootstrapACLs (boolean: false) - This flag controls whether the Helm chart automatically enables ACLs within the Consul cluster. This requires both Consul servers and clients to be run within Kubernetes. Requires Consul v1.5+ and consul-k8s v0.8.0+.

• server - Values that configure running a Consul server within Kubernetes.

  • enabled (boolean: global.enabled) - If true, the chart will install all the resources necessary for a Consul server cluster. If you're running Consul externally and want agents within Kubernetes to join that cluster, this should probably be false.

  • image (string: global.image) - The name of the Docker image (including any tag) for the containers running Consul server agents.

  • replicas (integer: 3) -The number of server agents to run. This determines the fault tolerance of the cluster. Please see the deployment table for more information.

  • bootstrapExpect (integer: 3) - For new clusters, this is the number of servers to wait for before performing the initial leader election and bootstrap of the cluster. This must be less than or equal to server.replicas. This value is only used when bootstrapping new clusters, it has no effect during ongoing cluster maintenance.

  • storage (string: 10Gi) - This defines the disk size for configuring the servers' StatefulSet storage. For dynamically provisioned storage classes, this is the desired size. For manually defined persistent volumes, this should be set to the disk size of the attached volume.

  • storageClass (string: null) - The StorageClass to use for the servers' StatefulSet storage. It must be able to be dynamically provisioned if you want the storage to be automatically created. For example, to use Local storage classes, the PersistentVolumeClaims would need to be manually created. A null value will use the Kubernetes cluster's default StorageClass. If a default StorageClass does not exist, you will need to create one.

  • connect (boolean: true) - This will enable/disable Connect. Setting this to true will not automatically secure pod communication, this setting will only enable usage of the feature. Consul will automatically initialize a new CA and set of certificates. Additional Connect settings can be configured by setting the server.extraConfig value.

  • resources (string: null) - The resource requests (CPU, memory, etc.) for each of the server agents. This should be a multi-line string mapping directly to a Kubernetes ResourceRequirements object. If this isn't specified, then the pods won't request any specific amount of resources. Setting this is highly recommended.

    ```yaml
    # Resources are defined as a formatted multi-line string:
    resources: |
      requests:
        memory: "10Gi"
      limits:
       memory: "10Gi"
    ```
    

  • updatePartition (integer: 0) - This value is used to carefully control a rolling update of Consul server agents. This value specifies the partition for performing a rolling update. Please read the linked Kubernetes documentation for more information.

  • disruptionBudget - This configures the PodDisruptionBudget for the server cluster.

    • enabled (boolean: true) - This will enable/disable registering a PodDisruptionBudget for the server cluster. If this is enabled, it will only register the budget so long as the server cluster is enabled.

    • maxUnavailable (integer: null) - The maximum number of unavailable pods. By default, this will be automatically computed based on the server.replicas value to be (n/2)-1. If you need to set this to 0, you will need to add a --set 'server.disruptionBudget.maxUnavailable=0' flag to the helm chart installation command because of a limitation in the Helm templating language.

  • extraConfig (string: "{}") - A raw string of extra JSON configuration for Consul servers. This will be saved as-is into a ConfigMap that is read by the Consul server agents. This can be used to add additional configuration that isn't directly exposed by the chart.

    ```yaml
    # ExtraConfig values are formatted as a multi-line string:
    extraConfig: |
      {
        "log_level": "DEBUG"
      }
    ```
    This can also be set using Helm's `--set` flag (consul-helm v0.7.0 and later), using the following syntax:
    
    ```shell
    --set 'server.extraConfig="{"log_level": "DEBUG"}"'
    ```
    

  • extraVolumes (array: []) - A list of extra volumes to mount for server agents. This is useful for bringing in extra data that can be referenced by other configurations at a well known path, such as TLS certificates or Gossip encryption keys. The value of this should be a list of objects. Each object supports the following keys:

    • type (string: required) - Type of the volume, must be one of "configMap" or "secret". Case sensitive.

    • name (string: required) - Name of the configMap or secret to be mounted. This also controls the path that it is mounted to. The volume will be mounted to /config/userconfig/<name>.

    • load (boolean: false) - If true, then the agent will be configured to automatically load HCL/JSON configuration files from this volume with -config-dir. This defaults to false.

      extraVolumes:    
        -  type: "secret"
           name: "consul-certs"
           load: false        
      

  • affinity (string) - This value defines the affinity for server pods. It defaults to allowing only a single pod on each node, which minimizes risk of the cluster becoming unusable if a node is lost. If you need to run more pods per node (for example, testing on Minikube), set this value to null.

    ```yaml
    # Recommended default server affinity:
    affinity: |
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: {{ template "consul.name" . }}
                release: "{{ .Release.Name }}"
                component: server
          topologyKey: kubernetes.io/hostname
    ```
    

  • priorityClassName (string) - This value references an existing Kubernetes priorityClassName that can be assigned to server pods.

  • annotations (string) - This value defines additional annotations for server pods. This should be a formatted as a multi-line string.

    ```yaml
    annotations: |
      "sample/annotation1": "foo"
      "sample/annotation2": "bar"
    ```
    

• client - Values that configure running a Consul client on Kubernetes nodes.

  • enabled (boolean: global.enabled) - If true, the chart will install all the resources necessary for a Consul client on every Kubernetes node. This does not require server.enabled, since the agents can be configured to join an external cluster.

  • image (string: global.image) - The name of the Docker image (including any tag) for the containers running Consul client agents.

  • join (array<string>: null) - A list of valid -retry-join values. If this is null (default), then the clients will attempt to automatically join the server cluster running within Kubernetes. This means that with server.enabled set to true, clients will automatically join that cluster. If server.enabled is not true, then a value must be specified so the clients can join a valid cluster.

  • grpc (boolean: false) - If true, agents will enable their GRPC listener on port 8502 and expose it to the host. This will use slightly more resources, but is required for Connect.

  • resources (string: null) - The resource requests (CPU, memory, etc.) for each of the client agents. This should be a multi-line string mapping directly to a Kubernetes ResourceRequirements object. If this isn't specified, then the pods won't request any specific amount of resources.

    ```yaml
    # Resources are defined as a formatted multi-line string:
    resources: |
      requests:
        memory: "10Gi"
      limits:
        memory: "10Gi"
    ```
    

  • extraConfig (string: "{}") - A raw string of extra JSON configuration for Consul clients. This will be saved as-is into a ConfigMap that is read by the Consul agents. This can be used to add additional configuration that isn't directly exposed by the chart.

    ```yaml
    # ExtraConfig values are formatted as a multi-line string:
    extraConfig: |
      {
        "log_level": "DEBUG"
      }
    ```
    This can also be set using Helm's `--set` flag (consul-helm v0.7.0 and later), using the following syntax:
    
    ```shell
    --set 'client.extraConfig="{"log_level": "DEBUG"}"'
    ```
    

  • extraVolumes (array: []) - A list of extra volumes to mount for client agents. This is useful for bringing in extra data that can be referenced by other configurations at a well known path, such as TLS certificates or Gossip encryption keys. The value of this should be a list of objects. Each object supports the following keys:

    • type (string: required) - Type of the volume, must be one of "configMap" or "secret". Case sensitive.

    • name (string: required) - Name of the configMap or secret to be mounted. This also controls the path that it is mounted to. The volume will be mounted to /config/userconfig/<name>.

    • load (boolean: false) - If true, then the agent will be configured to automatically load HCL/JSON configuration files from this volume with -config-dir. This defaults to false.

      extraVolumes:    
        -  type: "secret"
           name: "consul-certs"
           load: false        
      

  • priorityClassName (string) - This value references an existing Kubernetes priorityClassName that can be assigned to client pods.

  • annotations (string) - This value defines additional annotations for client pods. This should be a formatted as a multi-line string.

    ```yaml
    annotations: |
      "sample/annotation1": "foo"
      "sample/annotation2": "bar"
    ```
    

• dns - Values that configure Consul DNS service.

  • enabled (boolean: global.enabled) - If true, a consul-dns service will be created that exposes port 53 for TCP and UDP to the running Consul agents (servers and clients). This can then be used to configure kube-dns. The Helm chart does not automatically configure kube-dns.

• syncCatalog - Values that configure the service sync process.

  • enabled (boolean: false) - If true, the chart will install all the resources necessary for the catalog sync process to run.

  • image (string: global.imageK8S) - The name of the Docker image (including any tag) for consul-k8s to run the sync program.

  • default (boolean: true) - If true, all valid services in K8S are synced by default. If false, the service must be annotated properly to sync. In either case an annotation can override the default.

  • toConsul (boolean: true) - If true, will sync Kubernetes services to Consul. This can be disabled to have a one-way sync.

  • toK8S (boolean: true) - If true, will sync Consul services to Kubernetes. This can be disabled to have a one-way sync.

  • k8sPrefix (string: "") - A prefix to prepend to all services registered in Kubernetes from Consul. This defaults to "" where no prefix is prepended; Consul services are synced with the same name to Kubernetes. (Consul -> Kubernetes sync only)

  • consulPrefix (string: "") - A prefix to prepend to all services registered in Consul from Kubernetes. This defaults to "" where no prefix is prepended. Service names within Kubernetes remain unchanged. (Kubernetes -> Consul sync only)

  • k8sTag (string: null) - An optional tag that is applied to all of the Kubernetes services that are synced into Consul. If nothing is set, this defaults to "k8s". (Kubernetes -> Consul sync only)

  • syncClusterIPServices (boolean: true) - If true, will sync Kubernetes ClusterIP services to Consul. This can be disabled to have the sync ignore ClusterIP-type services.

  • nodePortSyncType (string: ExternalFirst) - Configures the type of syncing that happens for NodePort services. The only valid options are: ExternalOnly, InternalOnly, and ExternalFirst. ExternalOnly will only use a node's ExternalIP address for the sync, otherwise the service will not be synced. InternalOnly uses the node's InternalIP address. ExternalFirst will preferentially use the node's ExternalIP address, but if it doesn't exist, it will use the node's InternalIP address instead.

  • aclSyncToken - references a Kubernetes secret that contains an existing Consul ACL token. This will provide the sync process the correct permissions. This is only needed if ACLs are enabled on the Consul cluster.

    • secretName (string: null) - The name of the Kubernetes secret. This defaults to null.

    • secretKey (string: null) - The key for the Kubernetes secret. This defaults to null.

• ui - Values that configure the Consul UI.

  • enabled (boolean: global.enabled) - If true, the UI will be enabled. This will only enable the UI, it doesn't automatically register any service for external access. The UI will only be enabled on server agents. If server.enabled is false, then this setting has no effect. To expose the UI in some way, you must configure ui.service.

  • service - This configures the Service resource registered for the Consul UI.

    • enabled (boolean: true) - This will enable/disable registering a Kubernetes Service for the Consul UI. This value only takes effect if ui.enabled is true and taking effect.

    • type (string: null) - The service type to register. This defaults to null which doesn't set an explicit service type, which typically is defaulted to "ClusterIP" by Kubernetes. The available service types are documented on the Kubernetes website.

• connectInject - Values that configure running the Connect injector.

  • enabled (boolean: false) - If true, the chart will install all the resources necessary for the Connect injector process to run. This will enable the injector but will require pods to opt-in with an annotation by default.

  • image (string: global.imageK8S) - The name of the Docker image (including any tag) for the consul-k8s binary.

  • default (boolean: false) - If true, the injector will inject the Connect sidecar into all pods by default. Otherwise, pods must specify the. injection annotation to opt-in to Connect injection. If this is true, pods can use the same annotation to explicitly opt-out of injection.

  • imageConsul (string: global.image) - The name of the Docker image (including any tag) for Consul. This is used for proxy service registration, Envoy configuration, etc.

  • imageEnvoy (string: "") - The name of the Docker image (including any tag) for the Envoy sidecar. envoy must be on the executable path within this image. This Envoy version must be compatible with the Consul version used by the injector. This defaults to letting the injector choose the Envoy image, which is usually envoy/envoy-alpine.

  • namespaceSelector (string: "") - A selector for restricting injection to only matching namespaces. By default all namespaces except the system namespace will have injection enabled.

  • certs - The certs section configures how the webhook TLS certs are configured. These are the TLS certs for the Kube apiserver communicating to the webhook. By default, the injector will generate and manage its own certs, but this requires the ability for the injector to update its own MutatingWebhookConfiguration. In a production environment, custom certs should probably be used. Configure the values below to enable this.

    • secretName (string: null) - secretName is the name of the Kubernetes secret that has the TLS certificate and private key to serve the injector webhook. If this is null, then the injector will default to its automatic management mode.

    • caBundle (string: "") - The PEM-encoded CA public certificate bundle for the TLS certificate served by the injector. This must be specified as a string and can't come from a secret because it must be statically configured on the Kubernetes MutatingAdmissionWebhook resource. This only needs to be specified if secretName is not null.

    • certName (string: "tls.crt") - The name of the certificate file within the secretName secret.

    • keyName (string: "tls.key") - The name of the private key for the certificate file within the secretName secret.

  • aclBindingRuleSelector (string: "serviceaccount.name!=default") - A selector for restricting automatic injection to only matching services based on their associated service account. By default, services using the default Kubernetes service account will not have a proxy injected.

  • centralConfig - Values that configure Consul's central configuration feature (requires Consul v1.5+ and consul-k8s v0.8.1+).

    • enabled (boolean: false) - Turns on the central configuration feature. Pods that have a Connect proxy injected will have their service automatically registered in this central configuration.

    • defaultProtocol (string: null) - If defined, this value will be used as the default protocol type for all services registered with the central configuration. This can be overridden by using the protocol annotation directly on any pod spec.

    • proxyDefaults (string: "{}") - This value is a raw json string that will be applied to all Connect proxy sidecar pods. It can include any valid configuration for the configured proxy.

      # proxyDefaults values are formatted as a multi-line string:
      proxyDefaults: |
        {
          "envoy_dogstatsd_url": "udp://127.0.0.1:9125"
        }  
      

Using the Helm Chart to deploy Consul Enterprise

You can also use this Helm chart to deploy Consul Enterprise by following a few extra steps.

Find the license file that you received in your welcome email. It should have the extension .hclic. You will use the contents of this file to create a Kubernetes secret before installing the Helm chart.

You can use the following commands to create the secret:

secret=$(cat 1931d1f4-bdfd-6881-f3f5-19349374841f.hclic)
kubectl create secret generic consul-ent-license --from-literal="key=${secret}"

-> Note: If you cannot find your .hclic file, please contact your sales team or Technical Account Manager.

In your values.yaml, change the value of global.image to one of the enterprise release tags.

global:
  image: "hashicorp/consul-enterprise:1.4.3-ent"

Add the name of the secret you just created to server.enterpriseLicense.

server:
  enterpriseLicense:
    secretName: "consul-ent-license"
    secretKey: "key"

Now run helm install:

$ helm install --wait --name consul -f ./values.yaml ./consul-helm

Once the cluster is up, you can verify the nodes are running Consul Enterprise by using the consul license get command.

First, forward your local port 8500 to the Consul servers so you can run consul commands locally against the Consul servers in Kubernetes:

$ kubectl port-forward service/consul-consul-server -n default 8500

In a separate tab, run the consul license get command (if using ACLs see below):

$ consul license get
License is valid
License ID: 1931d1f4-bdfd-6881-f3f5-19349374841f
Customer ID: b2025a4a-8fdd-f268-95ce-1704723b9996
Expires At: 2020-03-09 03:59:59.999 +0000 UTC
Datacenter: *
Package: premium
Licensed Features:
        Automated Backups
        Automated Upgrades
        Enhanced Read Scalability
        Network Segments
        Redundancy Zone
        Advanced Network Federation
$ consul members
Node                                       Address           Status  Type    Build      Protocol  DC   Segment
consul-consul-server-0                     10.60.0.187:8301  alive   server  1.4.3+ent  2         dc1  <all>
consul-consul-server-1                     10.60.1.229:8301  alive   server  1.4.3+ent  2         dc1  <all>
consul-consul-server-2                     10.60.2.197:8301  alive   server  1.4.3+ent  2         dc1  <all>

If you get an error:

Error getting license: invalid character 'r' looking for beginning of value

Then you have likely enabled ACLs. You need to specify your ACL token when running the license get command. First, get the ACL token:

$ kubectl get secrets/consul-consul-bootstrap-acl-token --template={{.data.token}} | base64 -D
4dae8373-b4d7-8009-9880-a796850caef9%

Now use the token when running the license get command:

$ consul license get -token=4dae8373-b4d7-8009-9880-a796850caef9
License is valid
License ID: 1931d1f4-bdfd-6881-f3f5-19349374841f
Customer ID: b2025a4a-8fdd-f268-95ce-1704723b9996
Expires At: 2020-03-09 03:59:59.999 +0000 UTC
Datacenter: *
Package: premium
Licensed Features:
        Automated Backups
        Automated Upgrades
        Enhanced Read Scalability
        Network Segments
        Redundancy Zone
        Advanced Network Federation

Helm Chart Examples

The below values.yaml results in a single server Consul cluster with a LoadBalancer to allow external access to the UI and API.

global:
  enabled: true

server:
  replicas: 1
  bootstrapExpect: 1

ui:
  service:
    type: LoadBalancer

The below values.yaml results in a three server Consul Enterprise cluster with 100GB of storage and automatic Connect injection for annotated pods in the "my-app" namespace.

Note, this would require a secret that contains the enterprise license key.

global:
  image: "hashicorp/consul-enterprise:1.4.2-ent"
  datacenter: dc1

server:
  replicas: 3
  bootstrapExpect: 3
  enterpriseLicense:
    secretName: "consul-license"
    secretKey: "key"
  storage: 100Gi
  connect: true

client:
  grpc: true

connectInject:
  enabled: true
  default: false
  namespaceSelector: "my-app"

Customizing the Helm Chart

Consul within Kubernetes is highly configurable and the Helm chart contains dozens of the most commonly used configuration options. If you need to extend the Helm chart with additional options, we recommend using a third-party tool, such as kustomize or ship.