consul/website/source/docs/guides/kubernetes-production-deploy.md

name: "Consul-Kubernetes Deployment Guide" content_length: 14 id: kubernetes-production-deploy layout: content_layout products_used: - Consul description: This guide covers the necessary steps to install and configure a new Consul cluster on Kubernetes. level: Advanced ___ This guide covers the necessary steps to install and configure a new Consul cluster on Kubernetes, as defined in the Consul Reference Architecture guide. By the end of this guide, you will be able to identify the installation prerequisites, customize the Helm chart to fit your environment requirements, and interact with your new Consul cluster.
~> You should have the following configured before starting this guide: Helm installed and configured locally, tiller running in the Kubernetes cluster, and the Kubernetes CLI configured. ## Configure Kubernetes Permissions to Deploy Consul Before deploying Consul, you will need to create a new Kubernetes service account with the correct permissions and to authenticate it on the command line. You will need Kubernetes operators permissions to create and modify policies, deploy services, access the Kubernetes dashboard, create secrets, and create RBAC objects. You can find documentation for RBAC and service accounts for the following cloud providers. - AKS - EKS - GCP Note, Consul can be deployed on any properly configured Kubernetes cluster in the cloud or on premises. Once you have a service account, you will also need to add a permission to deploy the helm chart. This is done with the clusterrolebinding method. sh $ kubectl create clusterrolebinding kubernetes-dashboard -n kube-system --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard Finally, you may need to create Kubernetes secrets to store Consul data. You can reference these secrets in the customized Helm chart values file. - If you have purchased Enterprise Consul, the enterprise license file should be used with the official image, hashicorp/consul-enterprise:1.5.0-ent.
- Enable encryption to secure gossip traffic within the Consul cluster. ~> Note, depending on your environment, the previous secrets may not be necessary.
## Configure Helm Chart Now that you have prepared your Kubernetes cluster, you can customize the Helm chart. First, you will need to download the latest official Helm chart. sh $ git clone https://github.com/hashicorp/consul-helm.git The consul-helm directory will contain a values.yaml file with example parameters. You can update this file to customize your Consul deployment. Below we detail some of the parameters you should customize and provide an example file, however you should consider your particular production needs when configuring your chart. ### Global Values The global values will affect all the other parameters in the chart. To enable all of the Consul components in the Helm chart, set enabled to true. This means servers, clients, Consul DNS, and the Consul UI will be installed with their defaults. You should also set the following global parameters based on your specific environment requirements. - image is the name and tag of the Consul Docker image.
- imagek8s is the name and tag of the Docker image for the consul-k8s binary.
- datacenter the name of your Consul datacenter.
- domain the domain Consul uses for DNS queries. For security, set the bootstrapACLs parameter to true. This will enable Kubernetes to initially setup Consul's ACL system. Read the Consul Helm chart documentation to review all the global parameters. ### Consul UI To enable the Consul web UI update the ui section to your values file and set enabled to true. Note, you can also set up a loadbalancer resource or other service type in Kubernetes to make it easier to access the UI.
### Consul Servers For production deployments, you will need to deploy 3 or 5 Consul servers for quorum and failure tolerance. For most deployments, 3 servers are adequate. In the server section set both replicas and bootstrapExpect to 3. This will deploy three servers and cause Consul to wait to perform leader election until all three are healthy. The resources will depend on your environment; in the example at the end of the guide, the resources are set for a large environment. #### Affinity To ensure the Consul servers are placed on different Kubernetes nodes, you will need to configure affinity. Otherwise, the failure of one Kubernetes node could cause the loss of multiple Consul servers, and result in quorum loss. By default, the example values.yaml has affinity configured correctly.
#### Enterprise License If you have an Enterprise license you should reference the Kubernetes secret in the enterpriseLicense parameter. Read the Consul Helm chart documentation to review all the server parameters ### Consul Clients A Consul client is deployed on every Kubernetes node, so you do not need to specify the number of clients for your deployments. You will need to specify resources and enable gRPC. The resources in the example at the end of this guide should be sufficient for most production scenarios since Consul clients are designed for horizontal scalability. Enabling grpc enables the GRPC listener on port 8502 and exposes it to the host. It is required to use Consul Connect. Read the Consul Helm chart documentation to review all the client parameters ### Consul Connect Injection Security Even though you enabled Consul server communication over Connect in the server section, you will also need to enable connectInject by setting enabled to true. In the connectInject section you will also configure security features. Enabling the default parameter will allow the injector to automatically inject the Connect sidecar into all pods. If you would prefer to manually annotate which pods to inject, you can set this to false. Setting the 'aclBindingRuleSelectorparameter toserviceaccount.name!=defaultensures that new services do not all receive the same token if you are only using a default service account. This setting is only necessary if you have enabled ACLs in the global section. Read more about the [Connect Inject parameters](https://www.consul.io/docs/platform/k8s/helm.html#v-connectinject). ## Complete Example Your finished values file should resemble the following example. For more complete descriptions of all the available parameters see thevalues.yamlfile provided with the Helm chart and the [reference documentation](https://www.consul.io/docs/platform/k8s/helm.html). ```yaml # Configure global settings in this section. global: # Enable all the components within this chart by default. enabled: true # Specify the Consul and consul-k8s images to use image: "consul:1.5.0" imagek8s: "hashicorp/consul-k8s:0.8.1" domain: consul datacenter: primarydc # Bootstrap ACLs within Consul. This is highly recommended. bootstrapACLs: true # Gossip encryption gossipEncryption: | secretName: "encrypt-key" secretKey: "key # Configure your Consul servers in this section. server: enabled: true connect: true # Specify three servers that wait till all are healthy to bootstrap the Consul cluster. replicas: 3 bootstrapExpect: 3 # Specify the resources that servers request for placement. These values will serve a large environment. resources: | requests: memory: "32Gi" cpu: "4" disk: "50Gi" limits: memory: "32Gi" cpu: "4" disk: "50Gi" # If using Enterprise, reference the Kubernetes secret that holds your license here enterpriseLicense: secretName: "consul-license" secretKey: "key" # Prevent Consul servers from co-location on Kubernetes nodes. affinity: | podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app: {{ template "consul.name" . }} release: "{{ .Release.Name }}" component: server topologyKey: kubernetes.io/hostname # Configure Consul clients in this section client: enabled: true # Specify the resources that clients request for deployment. resources: | requests: memory: "8Gi" cpu: "2" disk: "15Gi" limits: memory: "8Gi" cpu: "2" disk: "15Gi" grpc: true # Enable and configure the Consul UI. ui: enabled: true # Configure security for Consul Connect pod injection connectInject: enabled: true default: true namespaceSelector: "my-namespace" aclBindingRuleSelector: “serviceaccount.name!=default” ``` ## Deploy Consul Now that you have customized thevalues.ymlfile, you can deploy Consul with Helm. This should only take a few minutes. The Consul pods should appear in the Kubernetes dashboard immediately and you can monitor the deployment process there. ```sh $ helm install ./consul-helm -f values.yaml ``` To check the deployment process on the command line you can usekubectl. ```sh $ kubectl get pods ``` ## Summary In this guide, you configured Consul, using the Helm chart, for a production environment. This involved ensuring that your cluster had a properly distributed server cluster, specifying enough resources for your agents, securing the cluster with ACLs and gossip encryption, and enabling other Consul functionality including Connect and the Consul UI. Now you can interact with your Consul cluster through the UI or CLI. If you exposed the UI using a load balancer it will be available at the LoadBalancer IngressIP address andPort` that is output from the following command. Note, you will need to replace consul server with the server name from your cluster. sh $ kubectl describe services consul-server To access the Consul CLI, open a terminal session using the Kubernetes CLI. sh $ kubectl exec <pod name> -it /bin/ash To learn more about how to interact with your Consul cluster or use it for service discovery, configuration or segmentation, try one of Learn’s Operations or Development tracks. Follow the Security and Networking track to learn more about securing your Consul cluster.