mirror of https://github.com/status-im/consul.git
581 lines
20 KiB
Plaintext
581 lines
20 KiB
Plaintext
---
|
||
layout: docs
|
||
page_title: How does Consul Service Mesh Work on Kubernetes?
|
||
description: >-
|
||
An injection annotation allows Consul to automatically deploy sidecar proxies on Kubernetes pods, enabling Consul's service mesh for containers running on k8s. Learn how to configure sidecars, enable services with multiple ports, change default injection settings.
|
||
---
|
||
|
||
# How does Consul Service Mesh Work on Kubernetes?
|
||
|
||
[Consul Service Mesh](/docs/connect) is a feature built into to Consul that enables
|
||
automatic service-to-service authorization and connection encryption across
|
||
your Consul services. Consul Service Mesh can be used with Kubernetes to secure pod
|
||
communication with other pods and external Kubernetes services. "Consul Connect" refers to the service mesh functionality within Consul and is used interchangeably with the name
|
||
"Consul Service Mesh."
|
||
|
||
The Connect sidecar running Envoy can be automatically injected into pods in
|
||
your cluster, making configuration for Kubernetes automatic.
|
||
This functionality is provided by the
|
||
[consul-k8s project](https://github.com/hashicorp/consul-k8s) and can be
|
||
automatically installed and configured using the
|
||
[Consul Helm chart](/docs/k8s/installation/install#helm-chart-installation).
|
||
|
||
## Usage
|
||
|
||
When the
|
||
[Connect injector is installed](/docs/k8s/connect#installation-and-configuration),
|
||
the Connect sidecar can be automatically added to all pods. This sidecar can both
|
||
accept and establish connections using Connect, enabling the pod to communicate
|
||
to clients and dependencies exclusively over authorized and encrypted
|
||
connections.
|
||
|
||
-> **Note:** The examples in this section are valid and use
|
||
publicly available images. If you've installed the Connect injector, feel free
|
||
to run the examples in this section to try Connect with Kubernetes.
|
||
Please note the documentation below this section on how to properly install
|
||
and configure the Connect injector.
|
||
|
||
### Accepting Inbound Connections
|
||
|
||
An example Deployment is shown below with Connect enabled to accept inbound
|
||
connections. Notice that the Deployment would still be fully functional without
|
||
Connect. Minimal to zero modifications are required to enable Connect in Kubernetes.
|
||
Notice also that even though we're using a Deployment here, the same configuration
|
||
would work on a Pod, a StatefulSet, or a DaemonSet.
|
||
|
||
This Deployment specification starts a server that responds to any
|
||
HTTP request with the static text "hello world".
|
||
|
||
-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes
|
||
service is **required** to run services on the Consul Service Mesh.
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
# This name will be the service name in Consul.
|
||
name: static-server
|
||
spec:
|
||
selector:
|
||
app: static-server
|
||
ports:
|
||
- protocol: TCP
|
||
port: 80
|
||
targetPort: 8080
|
||
---
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: static-server
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: static-server
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: static-server
|
||
template:
|
||
metadata:
|
||
name: static-server
|
||
labels:
|
||
app: static-server
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
spec:
|
||
containers:
|
||
- name: static-server
|
||
image: hashicorp/http-echo:latest
|
||
args:
|
||
- -text="hello world"
|
||
- -listen=:8080
|
||
ports:
|
||
- containerPort: 8080
|
||
name: http
|
||
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
|
||
serviceAccountName: static-server
|
||
```
|
||
|
||
The only change for Connect is the addition of the
|
||
`consul.hashicorp.com/connect-inject` annotation. This enables injection
|
||
for the Pod in this Deployment. The injector can also be
|
||
[configured](/docs/k8s/connect#installation-and-configuration)
|
||
to automatically inject unless explicitly disabled, but the default
|
||
installation requires opt-in using the annotation shown above.
|
||
|
||
~> **A common mistake** is to set the annotation on the Deployment or
|
||
other resource. Ensure that the injector annotations are specified on
|
||
the _pod specification template_ as shown above.
|
||
|
||
This will start a sidecar proxy that listens on port `20000` registered
|
||
with Consul and proxies valid inbound connections to port 8080 in the pod.
|
||
To establish a connection to the pod using Connect, a client must use another Connect
|
||
proxy. The client Connect proxy will use Consul service discovery to find
|
||
all available upstream proxies and their public ports.
|
||
|
||
In the example above, the server is listening on `:8080`.
|
||
By default, the Consul Service Mesh runs in [transparent proxy](/docs/connect/transparent-proxy) mode.
|
||
This means that even though the server binds to all interfaces,
|
||
the inbound and outbound connections will automatically go through to the sidecar proxy.
|
||
It also allows you to use Kubernetes DNS like you normally would without the
|
||
Consul Service Mesh.
|
||
|
||
-> **Note:** As of consul `v1.10.0`, consul-k8s `v0.26.0` and Consul Helm `v0.32.0`,
|
||
all Consul Service Mesh services will run with transparent proxy enabled by default. Running with transparent
|
||
proxy will enforce all inbound and outbound traffic to go through the Envoy proxy.
|
||
|
||
The service name registered in Consul will be set to the name of the Kubernetes service
|
||
associated with the Pod. This can be customized with the `consul.hashicorp.com/connect-service`
|
||
annotation. If using ACLs, this name must be the same as the Pod's `ServiceAccount` name.
|
||
|
||
### Connecting to Connect-Enabled Services
|
||
|
||
The example Deployment specification below configures a Deployment that is capable
|
||
of establishing connections to our previous example "static-server" service. The
|
||
connection to this static text service happens over an authorized and encrypted
|
||
connection via Connect.
|
||
|
||
-> **Note:** As of consul-k8s `v0.26.0` and Consul Helm `v0.32.0`, having a Kubernetes
|
||
Service is **required** to run services on the Consul Service Mesh.
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
# This name will be the service name in Consul.
|
||
name: static-client
|
||
spec:
|
||
selector:
|
||
app: static-client
|
||
ports:
|
||
- port: 80
|
||
---
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: static-client
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: static-client
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: static-client
|
||
template:
|
||
metadata:
|
||
name: static-client
|
||
labels:
|
||
app: static-client
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
spec:
|
||
containers:
|
||
- name: static-client
|
||
image: curlimages/curl:latest
|
||
# Just spin & wait forever, we'll use `kubectl exec` to demo
|
||
command: ['/bin/sh', '-c', '--']
|
||
args: ['while true; do sleep 30; done;']
|
||
# If ACLs are enabled, the serviceAccountName must match the Consul service name.
|
||
serviceAccountName: static-client
|
||
```
|
||
|
||
By default when ACLs are enabled or when ACLs default policy is `allow`,
|
||
Consul will automatically configure proxies with all upstreams from the same datacenter.
|
||
When ACLs are enabled with default `deny` policy,
|
||
you must supply an [intention](/docs/connect/intentions) to tell Consul which upstream you need to talk to.
|
||
|
||
When upstreams are specified explicitly with the
|
||
[`consul.hashicorp.com/connect-service-upstreams` annotation](/docs/k8s/annotations-and-labels#consul-hashicorp-com-connect-service-upstreams),
|
||
the injector will also set environment variables `<NAME>_CONNECT_SERVICE_HOST`
|
||
and `<NAME>_CONNECT_SERVICE_PORT` in every container in the Pod for every defined
|
||
upstream. This is analogous to the standard Kubernetes service environment variables, but
|
||
point instead to the correct local proxy port to establish connections via
|
||
Connect.
|
||
|
||
We can verify access to the static text server using `kubectl exec`.
|
||
Because transparent proxy is enabled by default,
|
||
we use Kubernetes DNS to connect to our desired upstream.
|
||
|
||
```shell-session
|
||
$ kubectl exec deploy/static-client -- curl --silent http://static-server/
|
||
"hello world"
|
||
```
|
||
|
||
We can control access to the server using [intentions](/docs/connect/intentions).
|
||
If you use the Consul UI or [CLI](/commands/intention/create) to
|
||
create a deny [intention](/docs/connect/intentions) between
|
||
"static-client" and "static-server", connections are immediately rejected
|
||
without updating either of the running pods. You can then remove this
|
||
intention to allow connections again.
|
||
|
||
```shell-session
|
||
$ kubectl exec deploy/static-client -- curl --silent http://static-server/
|
||
command terminated with exit code 52
|
||
```
|
||
|
||
### Kubernetes Pods with Multiple ports
|
||
To configure a pod with multiple ports to be a part of the service mesh and receive and send service mesh traffic, you
|
||
will need to add configuration so that a Consul service can be registered per port. This is because services in Consul
|
||
currently support a single port per service instance.
|
||
|
||
In the following example, suppose we have a pod which exposes 2 ports, `8080` and `9090`, both of which will need to
|
||
receive service mesh traffic.
|
||
|
||
First, decide on the names for the two Consul services that will correspond to those ports. In this example, the user
|
||
chooses the names `web` for `8080` and `web-admin` for `9090`.
|
||
|
||
Create two service accounts for `web` and `web-admin`:
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: web
|
||
---
|
||
apiVersion: v1
|
||
kind: ServiceAccount
|
||
metadata:
|
||
name: web-admin
|
||
```
|
||
Create two Service objects for `web` and `web-admin`:
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: web
|
||
spec:
|
||
selector:
|
||
app: web
|
||
ports:
|
||
- protocol: TCP
|
||
port: 80
|
||
targetPort: 8080
|
||
---
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: web-admin
|
||
spec:
|
||
selector:
|
||
app: web
|
||
ports:
|
||
- protocol: TCP
|
||
port: 80
|
||
targetPort: 9090
|
||
```
|
||
`web` will target `containerPort` `8080` and select pods labeled `app: web`. `web-admin` will target `containerPort`
|
||
`9090` and will also select the same pods.
|
||
|
||
~> Kubernetes 1.24+ only
|
||
In Kubernetes 1.24+ you need to [create a Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/#service-account-token-secrets) for each multi-port service that references the ServiceAccount, and the Kubernetes secret must have the same name as the ServiceAccount:
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Secret
|
||
metadata:
|
||
name: web
|
||
annotations:
|
||
kubernetes.io/service-account.name: web
|
||
type: kubernetes.io/service-account-token
|
||
---
|
||
apiVersion: v1
|
||
kind: Secret
|
||
metadata:
|
||
name: web-admin
|
||
annotations:
|
||
kubernetes.io/service-account.name: web-admin
|
||
type: kubernetes.io/service-account-token
|
||
```
|
||
|
||
Create a Deployment with any chosen name, and use the following annotations:
|
||
```yaml
|
||
consul.hashicorp.com/connect-inject: true
|
||
consul.hashicorp.com/transparent-proxy: false
|
||
consul.hashicorp.com/connect-service: web,web-admin
|
||
consul.hashicorp.com/connect-service-port: 8080,9090
|
||
```
|
||
Note that the order the ports are listed in the same order as the service names, i.e. the first service name `web`
|
||
corresponds to the first port, `8080`, and the second service name `web-admin` corresponds to the second port, `9090`.
|
||
|
||
The service account on the pod spec for the deployment should be set to the first service name `web`:
|
||
```yaml
|
||
serviceAccountName: web
|
||
```
|
||
|
||
For reference, the full deployment example could look something like the following:
|
||
```yaml
|
||
apiVersion: apps/v1
|
||
kind: Deployment
|
||
metadata:
|
||
name: web
|
||
spec:
|
||
replicas: 1
|
||
selector:
|
||
matchLabels:
|
||
app: web
|
||
template:
|
||
metadata:
|
||
name: web
|
||
labels:
|
||
app: web
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
'consul.hashicorp.com/transparent-proxy': 'false'
|
||
'consul.hashicorp.com/connect-service': 'web,web-admin'
|
||
'consul.hashicorp.com/connect-service-port': '8080,9090'
|
||
spec:
|
||
containers:
|
||
- name: web
|
||
image: hashicorp/http-echo:latest
|
||
args:
|
||
- -text="hello world"
|
||
- -listen=:8080
|
||
ports:
|
||
- containerPort: 8080
|
||
name: http
|
||
- name: web-admin
|
||
image: hashicorp/http-echo:latest
|
||
args:
|
||
- -text="hello world from 9090"
|
||
- -listen=:9090
|
||
ports:
|
||
- containerPort: 9090
|
||
name: http
|
||
serviceAccountName: web
|
||
```
|
||
|
||
After deploying the `web` application, you can test service mesh connections by deploying the `static-client`
|
||
application with the configuration in the [previous section](#connecting-to-connect-enabled-services) and add the
|
||
following annotation to the pod template on `static-client`:
|
||
```yaml
|
||
consul.hashicorp.com/connect-service-upstreams: "web:1234,web-admin:2234"
|
||
```
|
||
|
||
If you exec on to a static-client pod, using a command like:
|
||
```shell-session
|
||
$ kubectl exec -it static-client-5bd667fbd6-kk6xs -- /bin/sh
|
||
```
|
||
you can then run:
|
||
```shell-session
|
||
$ curl localhost:1234
|
||
```
|
||
to see the output `hello world` and run:
|
||
```shell-session
|
||
$ curl localhost:2234
|
||
```
|
||
to see the output `hello world from 9090`.
|
||
|
||
The way this works is that a Consul service instance is being registered per port on the Pod, so there are 2 Consul
|
||
services in this case. An additional Envoy sidecar proxy and `connect-init` init container are also deployed per port in
|
||
the Pod. So the upstream configuration can use the individual service names to reach each port as seen in the example.
|
||
|
||
|
||
#### Caveats for Multi-port Pods
|
||
* Transparent proxy is not supported for multi-port Pods.
|
||
* Metrics and metrics merging is not supported for multi-port Pods.
|
||
* Upstreams will only be set on the first service's Envoy sidecar proxy for the pod.
|
||
* This means that ServiceIntentions from a multi-port pod to elsewhere, will need to use the first service's name,
|
||
`web` in the example above to accept connections from either `web` or `web-admin`. ServiceIntentions from elsewhere
|
||
to a multi-port pod can use the individual service names within the multi-port Pod.
|
||
* Health checking is done on a per-Pod basis, so if any Kubernetes health checks (like readiness, liveness, etc) are
|
||
failing for any container on the Pod, the entire Pod is marked unhealthy, and any Consul service referencing that Pod
|
||
will also be marked as unhealthy. So, if `web` has a failing health check, `web-admin` would also be marked as
|
||
unhealthy for service mesh traffic.
|
||
|
||
## Installation and Configuration
|
||
|
||
The Connect sidecar proxy is injected via a
|
||
[mutating admission webhook](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#admission-webhooks)
|
||
provided by the
|
||
[consul-k8s project](https://github.com/hashicorp/consul-k8s).
|
||
This enables the automatic pod mutation shown in the usage section above.
|
||
Installation of the mutating admission webhook is automated using the
|
||
[Helm chart](/docs/k8s/installation/install).
|
||
|
||
To install the Connect injector, enable the Connect injection feature using
|
||
[Helm values](/docs/k8s/helm#configuration-values) and
|
||
upgrade the installation using `helm upgrade` for existing installs or
|
||
`helm install` for a fresh install.
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
|
||
controller:
|
||
enabled: true
|
||
```
|
||
|
||
This will configure the injector to inject when the
|
||
[injection annotation](#consul-hashicorp-com-connect-inject)
|
||
is set to `true`. Other values in the Helm chart can be used to limit the namespaces
|
||
the injector runs in, enable injection by default, and more.
|
||
|
||
### Verifying the Installation
|
||
|
||
To verify the installation, run the
|
||
["Accepting Inbound Connections"](/docs/k8s/connect#accepting-inbound-connections)
|
||
example from the "Usage" section above. After running this example, run
|
||
`kubectl get pod static-server --output yaml`. In the raw YAML output, you should
|
||
see injected Connect containers and an annotation
|
||
`consul.hashicorp.com/connect-inject-status` set to `injected`. This
|
||
confirms that injection is working properly.
|
||
|
||
If you do not see this, then use `kubectl logs` against the injector pod
|
||
and note any errors.
|
||
|
||
### Controlling Injection Via Annotation
|
||
|
||
By default, the injector will inject only when the
|
||
[injection annotation](#consul-hashicorp-com-connect-inject)
|
||
on the pod (not the deployment) is set to `true`:
|
||
|
||
```yaml
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
```
|
||
|
||
### Injection Defaults
|
||
|
||
If you wish for the injector to always inject, you can set the default to `true`
|
||
in the Helm chart:
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
default: true
|
||
```
|
||
|
||
You can then exclude specific pods via annotation:
|
||
|
||
```yaml
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'false'
|
||
```
|
||
|
||
### Controlling Injection Via Namespace
|
||
|
||
You can control which Kubernetes namespaces are allowed to be injected via
|
||
the `k8sAllowNamespaces` and `k8sDenyNamespaces` keys:
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
k8sAllowNamespaces: ['*']
|
||
k8sDenyNamespaces: []
|
||
```
|
||
|
||
In the default configuration (shown above), services from all namespaces are allowed
|
||
to be injected. Whether or not they're injected depends on the value of `connectInject.default`
|
||
and the `consul.hashicorp.com/connect-inject` annotation.
|
||
|
||
If you wish to only enable injection in specific namespaces, you can list only those
|
||
namespaces in the `k8sAllowNamespaces` key. In the configuration below
|
||
only the `my-ns-1` and `my-ns-2` namespaces will be enabled for injection.
|
||
All other namespaces will be ignored, even if the connect inject [annotation](#consul-hashicorp-com-connect-inject)
|
||
is set.
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
|
||
k8sDenyNamespaces: []
|
||
```
|
||
|
||
If you wish to enable injection in every namespace _except_ specific namespaces, you can
|
||
use `*` in the allow list to allow all namespaces and then specify the namespaces to exclude in the deny list:
|
||
|
||
```yaml
|
||
connectInject:
|
||
enabled: true
|
||
k8sAllowNamespaces: ['*']
|
||
k8sDenyNamespaces: ['no-inject-ns-1', 'no-inject-ns-2']
|
||
```
|
||
|
||
-> **NOTE:** The deny list takes precedence over the allow list. If a namespace
|
||
is listed in both lists, it will **not** be synced.
|
||
|
||
~> **NOTE:** The `kube-system` and `kube-public` namespaces will never be injected.
|
||
|
||
### Consul Enterprise Namespaces
|
||
|
||
Consul Enterprise 1.7+ supports Consul namespaces. When Kubernetes pods are registered
|
||
into Consul, you can control which Consul namespace they are registered into.
|
||
|
||
There are three options available:
|
||
|
||
1. **Single Destination Namespace** – Register all Kubernetes pods, regardless of namespace,
|
||
into the same Consul namespace.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
connectInject:
|
||
enabled: true
|
||
consulNamespaces:
|
||
consulDestinationNamespace: 'my-consul-ns'
|
||
```
|
||
|
||
-> **NOTE:** If the destination namespace does not exist we will create it.
|
||
|
||
1. **Mirror Namespaces** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes namespace.
|
||
For example, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `ns-1`.
|
||
If a mirrored namespace does not exist in Consul, it will be created.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
connectInject:
|
||
enabled: true
|
||
consulNamespaces:
|
||
mirroringK8S: true
|
||
```
|
||
|
||
1. **Mirror Namespaces With Prefix** - Register each Kubernetes pod into a Consul namespace with the same name as its Kubernetes
|
||
namespace **with a prefix**.
|
||
For example, given a prefix `k8s-`, pod `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `k8s-ns-1`.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
connectInject:
|
||
enabled: true
|
||
consulNamespaces:
|
||
mirroringK8S: true
|
||
mirroringK8SPrefix: 'k8s-'
|
||
```
|
||
|
||
### Consul Enterprise Namespace Upstreams
|
||
|
||
When [transparent proxy](/docs/connect/transparent-proxy) is enabled and ACLs are disabled,
|
||
the upstreams will be configured automatically across Consul namespaces.
|
||
When ACLs are enabled, you must configure it by specifying an [intention](/docs/connect/intentions),
|
||
allowing services across Consul namespaces to talk to each other.
|
||
|
||
If you wish to specify an upstream explicitly via the `consul.hashicorp.com/connect-service-upstreams` annotation,
|
||
use the format `[service-name].[namespace]:[port]:[optional datacenter]`:
|
||
|
||
```yaml
|
||
annotations:
|
||
'consul.hashicorp.com/connect-inject': 'true'
|
||
'consul.hashicorp.com/connect-service-upstreams': '[service-name].[namespace]:[port]:[optional datacenter]'
|
||
```
|
||
|
||
See [consul.hashicorp.com/connect-service-upstreams](#consul-hashicorp-com-connect-service-upstreams) for more details.
|
||
|
||
-> **Note:** When you specify upstreams via an upstreams annotation, you will need to use
|
||
`localhost:<port>` with the port from the upstreams annotation instead of KubeDNS to connect to your upstream
|
||
application.
|