consul/website/content/docs/k8s/connect/index.mdx

705 lines
24 KiB
Plaintext
Raw Normal View History

---
2020-04-07 18:55:19 +00:00
layout: docs
2022-09-14 22:48:49 +00:00
page_title: How does Consul Service Mesh Work on Kubernetes?
2020-04-07 18:55:19 +00:00
description: >-
2022-09-16 15:28:32 +00:00
An injection annotation allows Consul to automatically deploy sidecar proxies on Kubernetes pods, enabling Consul's service mesh for containers running on k8s. Learn how to configure sidecars, enable services with multiple ports, change default injection settings.
---
2022-09-14 22:48:49 +00:00
# How does Consul Service Mesh Work on Kubernetes?
Consul service mesh automates service-to-service authorization and encryption across your Consul services. You can use service mesh in Kubernetes-orchestrated networks to secure communication between pods as well as communication between pods and external Kubernetes services.
## Workflow
Consul service mesh is enabled by default when you install Consul on Kubernetes using the Consul Helm chart. Consul also automatically injects sidecars into the pods in your clusters that run Envoy. These sidecar proxies, called Consul dataplanes, are enabled when `connectInject.default` is set to `false` in the Helm chart. Refer to the following documentation for additional information about these concepts:
- [Installation and Configuration](#installation-and-configuration) in this topic
- [Consul Helm chart reference](/consul/docs/k8s/helm)
- [Simplified Service Mesh with Consul Dataplane](/consul/docs/connect/dataplane)
If `connectInject.default` is set to `false` or you want to explicitly enable service mesh sidecar proxy injection for a specific deployment, add the `consul.hashicorp.com/connect-inject` annotation to the pod specification template and set it to `true` when connecting services to the mesh.
### Service names
2023-07-07 17:03:28 +00:00
When the service is onboarded, the name registered in Consul is set to the name of the Kubernetes Service associated with the Pod. You can use the [`consul.hashicorp.com/connect-service` annotation](/consul/docs/k8s/annotations-and-labels#consul-hashicorp-com-connect-service) to specify a custom name for the service, but if ACLs are enabled then the name of the service registered in Consul must match the Pod's `ServiceAccount` name.
### Transparent proxy mode
By default, the Consul service mesh runs in transparent proxy mode. This mode forces inbound and outbound traffic through the sidecar proxy even though the service binds to all interfaces. Transparent proxy infers the location of upstream services using Consul service intentions, and also allows you to use Kubernetes DNS as you normally would for your workloads.
2023-07-07 17:03:28 +00:00
When transparent proxy mode is enabled, all service-to-service traffic is required to use mTLS. When onboarding new services to service mesh, your network may have mixed mTLS and non-mTLS traffic, which can result in broken service-to-service communication. You can temporarily enable permissive mTLS mode during the onboarding process so that existing mesh services can accept traffic from services that are not yet fully onboarded. Permissive mTLS enables sidecar proxies to access both mTLS and non-mTLS traffic. Refer to [Onboard mesh services in transparent proxy mode](/consul/docs/k8s/connect/onboarding-tproxy-mode) for additional information.
### Kubernetes service mesh workload scenarios
2023-07-07 17:03:28 +00:00
-> **Note:** A Kubernetes Service is required in order to register services on the Consul service mesh. Consul monitors the lifecyle of the Kubernetes Service and its service instances using the service object. In addition, the Kubernetes service is used to register and de-register the service from Consul's catalog.
2023-07-07 17:03:28 +00:00
The following configurations are examples for registering workloads on Kubernetes into Consul's service mesh in different scenarios. Each scenario provides an example Kubernetes manifest to demonstrate how to use Consul's service mesh with a specific Kubernetes workload type.
- [Kubernetes Pods running as a deployment](#kubernetes-pods-running-as-a-deployment)
- [Connecting to mesh-enabled Services](#connecting-to-mesh-enabled-services)
- [Kubernetes Jobs](#kubernetes-jobs)
2023-07-07 17:03:28 +00:00
- [Kubernetes Pods with multiple ports](#kubernetes-pods-with-multiple-ports)
#### Kubernetes Pods running as a deployment
The following example shows a Kubernetes configuration that specifically enables service mesh connections for the `static-server` service. Consul starts and registers a sidecar proxy that listens on port 20000 by default and proxies valid inbound connections to port 8080.
<CodeBlockConfig filename="static-server.yaml">
```yaml
apiVersion: v1
kind: Service
metadata:
# This name will be the service name in Consul.
name: static-server
spec:
selector:
app: static-server
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
2018-10-08 15:24:25 +00:00
name: static-server
spec:
replicas: 1
selector:
matchLabels:
app: static-server
template:
metadata:
name: static-server
labels:
app: static-server
annotations:
'consul.hashicorp.com/connect-inject': 'true'
spec:
containers:
- name: static-server
image: hashicorp/http-echo:latest
args:
- -text="hello world"
- -listen=:8080
ports:
- containerPort: 8080
name: http
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
serviceAccountName: static-server
```
</CodeBlockConfig>
2023-07-07 17:03:28 +00:00
To establish a connection to the upstream Pod using service mesh, a client must dial the upstream workload using a mesh proxy. The client mesh proxy will use Consul service discovery to find all available upstream proxies and their public ports.
2023-07-07 17:03:28 +00:00
#### Connecting to mesh-enabled Services
The example Deployment specification below configures a Deployment that is capable
2018-10-08 15:24:25 +00:00
of establishing connections to our previous example "static-server" service. The
connection to this static text service happens over an authorized and encrypted
connection via service mesh.
<CodeBlockConfig filename="static-client.yaml">
```yaml
apiVersion: v1
kind: Service
metadata:
# This name will be the service name in Consul.
name: static-client
spec:
selector:
app: static-client
ports:
- port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
2018-10-08 15:24:25 +00:00
name: static-client
spec:
replicas: 1
selector:
matchLabels:
app: static-client
template:
metadata:
name: static-client
labels:
app: static-client
annotations:
'consul.hashicorp.com/connect-inject': 'true'
spec:
containers:
- name: static-client
image: curlimages/curl:latest
# Just spin & wait forever, we'll use `kubectl exec` to demo
command: ['/bin/sh', '-c', '--']
args: ['while true; do sleep 30; done;']
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
serviceAccountName: static-client
```
</CodeBlockConfig>
By default when ACLs are enabled or when ACLs default policy is `allow`,
Consul will automatically configure proxies with all upstreams from the same datacenter.
When ACLs are enabled with default `deny` policy,
you must supply an [intention](/consul/docs/connect/intentions) to tell Consul which upstream you need to talk to.
2018-10-08 16:55:55 +00:00
When upstreams are specified explicitly with the
[`consul.hashicorp.com/connect-service-upstreams` annotation](/consul/docs/k8s/annotations-and-labels#consul-hashicorp-com-connect-service-upstreams),
the injector will also set environment variables `<NAME>_CONNECT_SERVICE_HOST`
and `<NAME>_CONNECT_SERVICE_PORT` in every container in the Pod for every defined
2018-10-09 16:30:37 +00:00
upstream. This is analogous to the standard Kubernetes service environment variables, but
2018-10-08 16:55:55 +00:00
point instead to the correct local proxy port to establish connections via
service mesh.
2018-10-08 16:55:55 +00:00
You can verify access to the static text server using `kubectl exec`.
Because transparent proxy is enabled by default,
use Kubernetes DNS to connect to your desired upstream.
2020-05-19 18:32:38 +00:00
```shell-session
$ kubectl exec deploy/static-client -- curl --silent http://static-server/
"hello world"
```
You can control access to the server using [intentions](/consul/docs/connect/intentions).
If you use the Consul UI or [CLI](/consul/commands/intention/create) to
deny communication between
2018-10-08 15:24:25 +00:00
"static-client" and "static-server", connections are immediately rejected
without updating either of the running pods. You can then remove this
intention to allow connections again.
2020-05-19 18:32:38 +00:00
```shell-session
$ kubectl exec deploy/static-client -- curl --silent http://static-server/
command terminated with exit code 52
```
#### Kubernetes Jobs
2023-07-07 17:03:28 +00:00
Kubernetes Jobs run pods that only make outbound requests to services on the mesh and successfully terminate when they are complete. In order to register a Kubernetes Job with the mesh, you must provide an integer value for the `consul.hashicorp.com/sidecar-proxy-lifecycle-shutdown-grace-period-seconds` annotation. Then, issue a request to the `http://127.0.0.1:20600/graceful_shutdown` API endpoint so that Kubernetes gracefully shuts down the `consul-dataplane` sidecar after the job is complete.
Below is an example Kubernetes manifest that deploys a job correctly.
<CodeBlockConfig filename="test-job.yaml">
```yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-job
namespace: default
---
apiVersion: v1
kind: Service
metadata:
name: test-job
namespace: default
spec:
selector:
app: test-job
ports:
- port: 80
---
apiVersion: batch/v1
kind: Job
metadata:
name: test-job
namespace: default
labels:
app: test-job
spec:
template:
metadata:
annotations:
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/sidecar-proxy-lifecycle-shutdown-grace-period-seconds': '5'
labels:
app: test-job
spec:
containers:
- name: test-job
image: alpine/curl:3.14
ports:
- containerPort: 80
command:
- /bin/sh
- -c
- |
echo "Started test job"
sleep 10
echo "Killing proxy"
curl --max-time 2 -s -f -XPOST http://127.0.0.1:20600/graceful_shutdown
sleep 10
echo "Ended test job"
serviceAccountName: test-job
restartPolicy: Never
```
</CodeBlockConfig>
Upon completing the job you should be able to verify that all containers are shut down within the pod.
```shell-session
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
test-job-49st7 0/2 Completed 0 3m55s
```
```shell-session
$ kubectl get job
NAME COMPLETIONS DURATION AGE
test-job 1/1 30s 4m31s
```
2023-07-07 17:03:28 +00:00
In addition, based on the logs emitted by the pod you can verify that the proxy was shut down before the Job completed.
```shell-session
$ kubectl logs test-job-49st7 -c test-job
Started test job
Killing proxy
Ended test job
```
2023-07-07 17:03:28 +00:00
#### Kubernetes Pods with multiple ports
To configure a pod with multiple ports to be a part of the service mesh and receive and send service mesh traffic, you
will need to add configuration so that a Consul service can be registered per port. This is because services in Consul
currently support a single port per service instance.
In the following example, suppose we have a pod which exposes 2 ports, `8080` and `9090`, both of which will need to
receive service mesh traffic.
First, decide on the names for the two Consul services that will correspond to those ports. In this example, the user
chooses the names `web` for `8080` and `web-admin` for `9090`.
Create two service accounts for `web` and `web-admin`:
<CodeBlockConfig filename="multiport-web-sa.yaml">
```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: web
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: web-admin
```
</CodeBlockConfig>
Create two Service objects for `web` and `web-admin`:
<CodeBlockConfig filename="multiport-web-svc.yaml">
```yaml
apiVersion: v1
kind: Service
metadata:
name: web
spec:
selector:
app: web
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
name: web-admin
spec:
selector:
app: web
ports:
- protocol: TCP
port: 80
targetPort: 9090
```
</CodeBlockConfig>
`web` will target `containerPort` `8080` and select pods labeled `app: web`. `web-admin` will target `containerPort`
`9090` and will also select the same pods.
~> Kubernetes 1.24+ only
In Kubernetes 1.24+ you need to [create a Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets) for each multi-port service that references the ServiceAccount, and the Kubernetes secret must have the same name as the ServiceAccount:
<CodeBlockConfig filename="multiport-web-secret.yaml">
```yaml
apiVersion: v1
kind: Secret
metadata:
name: web
annotations:
kubernetes.io/service-account.name: web
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
name: web-admin
annotations:
kubernetes.io/service-account.name: web-admin
type: kubernetes.io/service-account-token
```
</CodeBlockConfig>
Create a Deployment with any chosen name, and use the following annotations:
```yaml
annotations:
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/transparent-proxy': 'false'
'consul.hashicorp.com/connect-service': 'web,web-admin'
'consul.hashicorp.com/connect-service-port': '8080,9090'
```
Note that the order the ports are listed in the same order as the service names, i.e. the first service name `web`
corresponds to the first port, `8080`, and the second service name `web-admin` corresponds to the second port, `9090`.
The service account on the pod spec for the deployment should be set to the first service name `web`:
```yaml
serviceAccountName: web
```
2023-07-07 17:03:28 +00:00
The following deployment example demonstrates the required annotations for the manifest. In addition, the previous YAML manifests can also be combined into a single manifest for easier deployment.
<CodeBlockConfig filename="multiport-web.yaml">
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
name: web
labels:
app: web
annotations:
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/transparent-proxy': 'false'
'consul.hashicorp.com/connect-service': 'web,web-admin'
'consul.hashicorp.com/connect-service-port': '8080,9090'
spec:
containers:
- name: web
image: hashicorp/http-echo:latest
args:
- -text="hello world"
- -listen=:8080
ports:
- containerPort: 8080
name: http
- name: web-admin
image: hashicorp/http-echo:latest
args:
- -text="hello world from 9090"
- -listen=:9090
ports:
- containerPort: 9090
name: http
serviceAccountName: web
```
</CodeBlockConfig>
After deploying the `web` application, you can test service mesh connections by deploying the `static-client`
application with the configuration in the [previous section](#connecting-to-mesh-enabled-services) and add the
`consul.hashicorp.com/connect-service-upstreams: 'web:1234,web-admin:2234'` annotation to the pod template on `static-client`:
<CodeBlockConfig filename="multiport-static-client.yaml" lineNumbers highlight="33">
```yaml
apiVersion: v1
kind: Service
metadata:
# This name will be the service name in Consul.
name: static-client
spec:
selector:
app: static-client
ports:
- port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: static-client
spec:
replicas: 1
selector:
matchLabels:
app: static-client
template:
metadata:
name: static-client
labels:
app: static-client
annotations:
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/connect-service-upstreams': 'web:1234,web-admin:2234'
spec:
containers:
- name: static-client
image: curlimages/curl:latest
# Just spin & wait forever, we'll use `kubectl exec` to demo
command: ['/bin/sh', '-c', '--']
args: ['while true; do sleep 30; done;']
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
serviceAccountName: static-client
```
</CodeBlockConfig>
If you exec on to a static-client pod, using a command like:
```shell-session
$ kubectl exec -it static-client-5bd667fbd6-kk6xs -- /bin/sh
```
you can then run:
```shell-session
$ curl localhost:1234
```
to see the output `hello world` and run:
```shell-session
$ curl localhost:2234
```
to see the output `hello world from 9090`.
The way this works is that a Consul service instance is being registered per port on the Pod, so there are 2 Consul
services in this case. An additional Envoy sidecar proxy and `connect-init` init container are also deployed per port in
the Pod. So the upstream configuration can use the individual service names to reach each port as seen in the example.
#### Caveats for Multi-port Pods
* Transparent proxy is not supported for multi-port Pods.
* Metrics and metrics merging is not supported for multi-port Pods.
* Upstreams will only be set on the first service's Envoy sidecar proxy for the pod.
* This means that ServiceIntentions from a multi-port pod to elsewhere, will need to use the first service's name,
`web` in the example above to accept connections from either `web` or `web-admin`. ServiceIntentions from elsewhere
to a multi-port pod can use the individual service names within the multi-port Pod.
* Health checking is done on a per-Pod basis, so if any Kubernetes health checks (like readiness, liveness, etc) are
failing for any container on the Pod, the entire Pod is marked unhealthy, and any Consul service referencing that Pod
will also be marked as unhealthy. So, if `web` has a failing health check, `web-admin` would also be marked as
unhealthy for service mesh traffic.
## Installation and Configuration
The service mesh sidecar proxy is injected via a
[mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks)
call the connect injector provided by the
[consul-k8s project](https://github.com/hashicorp/consul-k8s).
This enables the automatic pod mutation shown in the usage section above.
Installation of the mutating admission webhook is automated using the
[Helm chart](/consul/docs/k8s/installation/install).
To install the connect injector, enable the connect injection feature using
[Helm values](/consul/docs/k8s/helm#configuration-values) and
upgrade the installation using `helm upgrade` for existing installs or
`helm install` for a fresh install.
```yaml
connectInject:
enabled: true
```
This will configure the injector to inject when the
2020-02-04 19:05:25 +00:00
[injection annotation](#consul-hashicorp-com-connect-inject)
is set to `true`. Other values in the Helm chart can be used to limit the namespaces
the injector runs in, enable injection by default, and more.
### Verifying the Installation
To verify the installation, run the
["Accepting Inbound Connections"](/consul/docs/k8s/connect#accepting-inbound-connections)
example from the "Usage" section above. After running this example, run
`kubectl get pod static-server --output yaml`. In the raw YAML output, you should
see connect injected containers and an annotation
`consul.hashicorp.com/connect-inject-status` set to `injected`. This
confirms that injection is working properly.
If you do not see this, then use `kubectl logs` against the injector pod
and note any errors.
2020-02-04 19:05:25 +00:00
### Controlling Injection Via Annotation
By default, the injector will inject only when the
[injection annotation](#consul-hashicorp-com-connect-inject)
on the pod (not the deployment) is set to `true`:
```yaml
annotations:
2020-04-06 20:27:35 +00:00
'consul.hashicorp.com/connect-inject': 'true'
2020-02-04 19:05:25 +00:00
```
### Injection Defaults
If you wish for the injector to always inject, you can set the default to `true`
in the Helm chart:
```yaml
connectInject:
enabled: true
default: true
```
You can then exclude specific pods via annotation:
```yaml
annotations:
2020-04-06 20:27:35 +00:00
'consul.hashicorp.com/connect-inject': 'false'
2020-02-04 19:05:25 +00:00
```
### Controlling Injection Via Namespace
You can control which Kubernetes namespaces are allowed to be injected via
the `k8sAllowNamespaces` and `k8sDenyNamespaces` keys:
```yaml
connectInject:
enabled: true
2020-04-06 20:27:35 +00:00
k8sAllowNamespaces: ['*']
2020-02-04 19:05:25 +00:00
k8sDenyNamespaces: []
```
In the default configuration (shown above), services from all namespaces are allowed
to be injected. Whether or not they're injected depends on the value of `connectInject.default`
and the `consul.hashicorp.com/connect-inject` annotation.
If you wish to only enable injection in specific namespaces, you can list only those
namespaces in the `k8sAllowNamespaces` key. In the configuration below
only the `my-ns-1` and `my-ns-2` namespaces will be enabled for injection.
All other namespaces will be ignored, even if the connect inject [annotation](#consul-hashicorp-com-connect-inject)
is set.
```yaml
connectInject:
enabled: true
2020-04-06 20:27:35 +00:00
k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
2020-02-04 19:05:25 +00:00
k8sDenyNamespaces: []
```
2020-04-06 20:27:35 +00:00
If you wish to enable injection in every namespace _except_ specific namespaces, you can
2020-02-04 19:05:25 +00:00
use `*` in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list:
```yaml
connectInject:
2020-02-04 19:05:25 +00:00
enabled: true
2020-04-06 20:27:35 +00:00
k8sAllowNamespaces: ['*']
k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']
2020-02-04 19:05:25 +00:00
```
-> **NOTE:** The deny list takes precedence over the allow list. If a namespace
2020-04-06 20:27:35 +00:00
is listed in both lists, it will **not** be synced.
2020-02-04 19:05:25 +00:00
~> **NOTE:** The `kube-system` and `kube-public` namespaces will never be injected.
### Consul Enterprise Namespaces
Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered
into Consul, you can control which Consul namespace they are registered into.
There are three options available:
2020-04-06 20:27:35 +00:00
1. **Single Destination Namespace** Register all Kubernetes pods, regardless of namespace,
into the same Consul namespace.
2020-02-04 19:05:25 +00:00
This can be configured with:
2020-02-04 19:05:25 +00:00
```yaml
global:
enableConsulNamespaces: true
2020-04-06 20:27:35 +00:00
connectInject:
enabled: true
consulNamespaces:
consulDestinationNamespace: 'my-consul-ns'
```
2020-02-04 19:05:25 +00:00
-> **NOTE:** If the destination namespace does not exist we will create it.
2020-02-04 19:05:25 +00:00
2020-04-06 20:27:35 +00:00
1. **Mirror Namespaces** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace.
For example, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `ns-1`.
If a mirrored namespace does not exist in Consul, it will be created.
2020-04-07 18:55:19 +00:00
This can be configured with:
```yaml
2020-02-04 19:05:25 +00:00
global:
enableConsulNamespaces: true
2020-04-07 18:55:19 +00:00
connectInject:
enabled: true
consulNamespaces:
mirroringK8S: true
```
2020-02-04 19:05:25 +00:00
2020-04-06 20:27:35 +00:00
1. **Mirror Namespaces With Prefix** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes
namespace **with a prefix**.
For example, given a prefix `k8s-`, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `k8s-ns-1`.
2020-02-04 19:05:25 +00:00
This can be configured with:
2020-04-06 20:27:35 +00:00
```yaml
global:
enableConsulNamespaces: true
2020-04-06 20:27:35 +00:00
connectInject:
enabled: true
consulNamespaces:
mirroringK8S: true
mirroringK8SPrefix: 'k8s-'
```
2020-02-04 19:05:25 +00:00
### Consul Enterprise Namespace Upstreams
2020-04-06 20:27:35 +00:00
When [transparent proxy](/consul/docs/connect/transparent-proxy) is enabled and ACLs are disabled,
the upstreams will be configured automatically across Consul namespaces.
When ACLs are enabled, you must configure it by specifying an [intention](/consul/docs/connect/intentions),
allowing services across Consul namespaces to talk to each other.
If you wish to specify an upstream explicitly via the `consul.hashicorp.com/connect-service-upstreams` annotation,
2020-02-04 19:05:25 +00:00
use the format `[service-name].[namespace]:[port]:[optional datacenter]`:
```yaml
annotations:
2020-04-06 20:27:35 +00:00
'consul.hashicorp.com/connect-inject': 'true'
'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'
2020-02-04 19:05:25 +00:00
```
See [consul.hashicorp.com/connect-service-upstreams](#consul-hashicorp-com-connect-service-upstreams) for more details.
-> **Note:** When you specify upstreams via an upstreams annotation, you will need to use
`localhost:<port>` with the port from the upstreams annotation instead of KubeDNS to connect to your upstream
application.