status-im
/
consul
mirror of https://github.com/status-im/consul.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
consul/website/content/docs/connect/cluster-peering/k8s.mdx

594 lines
19 KiB
Plaintext
Raw Blame History

---
layout: docs
page_title: Cluster Peering on Kubernetes
description: >-
If you use Consul on Kubernetes, learn how to enable cluster peering, create peering CRDs, and then manage peering connections in consul-k8s.
---
# Cluster Peering on Kubernetes
To establish a cluster peering connection on Kubernetes, you need to enable several pre-requisite values in the Helm chart and create custom resource definitions (CRDs) for each side of the peering.
The following Helm values are mandatory for cluster peering:
- [`global.tls.enabled = true`](/consul/docs/k8s/helm#v-global-tls-enabled)
- [`meshGateway.enabled = true`](/consul/docs/k8s/helm#v-meshgateway-enabled)
The following CRDs are used to create and manage a peering connection:
- `PeeringAcceptor`: Generates a peering token and accepts an incoming peering connection.
- `PeeringDialer`: Uses a peering token to make an outbound peering connection with the cluster that generated the token.
Peering connections, including both data plane and control plane traffic, is routed through mesh gateways.
As of Consul v1.14, you can also [implement service failovers and redirects to control traffic](/consul/docs/connect/l7-traffic) between peers.
> To learn how to peer clusters and connect services across peers in AWS Elastic Kubernetes Service (EKS) environments, complete the [Consul Cluster Peering on Kubernetes tutorial](/consul/tutorials/developer-mesh/cluster-peering-aws?utm_source=docs).
## Prerequisites
You must implement the following requirements to create and use cluster peering connections with Kubernetes:
- Consul v1.14.0 or later
- At least two Kubernetes clusters
- The installation must be running on Consul on Kubernetes version 1.0.0 or later
### Prepare for installation
Complete the following procedure after you have provisioned a Kubernetes cluster and set up your kubeconfig file to manage access to multiple Kubernetes clusters.
1. Use the `kubectl` command to export the Kubernetes context names and then set them to variables for future use. For more information on how to use kubeconfig and contexts, refer to the [Kubernetes docs on configuring access to multiple clusters](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/).
You can use the following methods to get the context names for your clusters:
- Use the `kubectl config current-context` command to get the context for the cluster you are currently in.
- Use the `kubectl config get-contexts` command to get all configured contexts in your kubeconfig file.
```shell-session
$ export CLUSTER1_CONTEXT=<CONTEXT for first Kubernetes cluster>
$ export CLUSTER2_CONTEXT=<CONTEXT for second Kubernetes cluster>
```
1. To establish cluster peering through Kubernetes, create a `values.yaml` file with the following Helm values. **NOTE:** Mesh Gateway replicas are defaulted to 1 replica, and could be adjusted using the `meshGateway.replicas` value for higher availability.
<CodeBlockConfig filename="values.yaml">
```yaml
global:
name: consul
image: "hashicorp/consul:1.14.1"
peering:
enabled: true
tls:
enabled: true
meshGateway:
enabled: true
```
</CodeBlockConfig>
### Install Consul on Kubernetes
Install Consul on Kubernetes by using the CLI to apply `values.yaml` to each cluster.
1. In `cluster-01`, run the following commands:
```shell-session
$ export HELM_RELEASE_NAME=cluster-01
```
```shell-session
$ helm install ${HELM_RELEASE_NAME} hashicorp/consul --create-namespace --namespace consul --version "1.0.1" --values values.yaml --set global.datacenter=dc1 --kube-context $CLUSTER1_CONTEXT
```
1. In `cluster-02`, run the following commands:
```shell-session
$ export HELM_RELEASE_NAME=cluster-02
```
```shell-session
$ helm install ${HELM_RELEASE_NAME} hashicorp/consul --create-namespace --namespace consul --version "1.0.1" --values values.yaml --set global.datacenter=dc2 --kube-context $CLUSTER2_CONTEXT
```
## Create a peering connection for Consul on Kubernetes
To peer Kubernetes clusters running Consul, you need to create a peering token on one cluster (`cluster-01`) and share
it with the other cluster (`cluster-02`). The generated peering token from `cluster-01` will include the addresses of
the servers for that cluster. The servers for `cluster-02` will use that information to dial the servers in
`cluster-01`. Complete the following steps to create the peer connection.
### Using mesh gateways for the peering connection
If the servers in `cluster-01` are not directly routable from the dialing cluster `cluster-02`, then you'll need to set up peering through mesh gateways.
1. In `cluster-01` apply the `Mesh` custom resource so the generated token will have the mesh gateway addresses which will be routable from the other cluster.
<CodeBlockConfig filename="mesh.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
name: mesh
spec:
peering:
peerThroughMeshGateways: true
```
</CodeBlockConfig>
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT apply -f mesh.yaml
```
1. In `cluster-02` apply the `Mesh` custom resource so that the servers for `cluster-02` will use their local mesh gateway to dial the servers for `cluster-01`.
<CodeBlockConfig filename="mesh.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: Mesh
metadata:
name: mesh
spec:
peering:
peerThroughMeshGateways: true
```
</CodeBlockConfig>
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply -f mesh.yaml
```
### Create a peering token
Peers identify each other using the `metadata.name` values you establish when creating the `PeeringAcceptor` and `PeeringDialer` CRDs.
1. In `cluster-01`, create the `PeeringAcceptor` custom resource.
<CodeBlockConfig filename="acceptor.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringAcceptor
metadata:
name: cluster-02 ## The name of the peer you want to connect to
spec:
peer:
secret:
name: "peering-token"
key: "data"
backend: "kubernetes"
```
</CodeBlockConfig>
1. Apply the `PeeringAcceptor` resource to the first cluster.
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT apply --filename acceptor.yaml
````
1. Save your peering token so that you can export it to the other cluster.
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT get secret peering-token --output yaml > peering-token.yaml
```
### Establish a peering connection between clusters
1. Apply the peering token to the second cluster.
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply --filename peering-token.yaml
```
1. In `cluster-02`, create the `PeeringDialer` custom resource.
<CodeBlockConfig filename="dialer.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringDialer
metadata:
name: cluster-01 ## The name of the peer you want to connect to
spec:
peer:
secret:
name: "peering-token"
key: "data"
backend: "kubernetes"
```
</CodeBlockConfig>
1. Apply the `PeeringDialer` resource to the second cluster.
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply --filename dialer.yaml
```
### Configure the mesh gateway mode for traffic between services
Mesh gateways are required for service-to-service traffic between peered clusters. By default, this will mean that a
service dialing another service in a remote peer will dial the remote mesh gateway to reach that service. If you would
like to configure the mesh gateway mode such that this traffic always leaves through the local mesh gateway, you can use the `ProxyDefaults` CRD.
1. In `cluster-01` apply the following `ProxyDefaults` CRD to configure the mesh gateway mode.
<CodeBlockConfig filename="proxy-defaults.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
meshGateway:
mode: local
```
</CodeBlockConfig>
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT apply -f proxy-defaults.yaml
```
1. In `cluster-02` apply the following `ProxyDefaults` CRD to configure the mesh gateway mode.
<CodeBlockConfig filename="proxy-defaults.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
meshGateway:
mode: local
```
</CodeBlockConfig>
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply -f proxy-defaults.yaml
```
### Export services between clusters
The examples described in this section demonstrate how to export a service named `backend`. You should change instances of `backend` in the example code to the name of the service you want to export.
1. For the service in `cluster-02` that you want to export, add the `"consul.hashicorp.com/connect-inject": "true"` annotation to your service's pods prior to deploying. The annotation allows the workload to join the mesh. It is highlighted in the following example:
<CodeBlockConfig filename="backend.yaml" highlight="37">
```yaml
# Service to expose backend
apiVersion: v1
kind: Service
metadata:
name: backend
spec:
selector:
app: backend
ports:
- name: http
protocol: TCP
port: 80
targetPort: 9090
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
---
# Deployment for backend
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
labels:
app: backend
spec:
replicas: 1
selector:
matchLabels:
app: backend
template:
metadata:
labels:
app: backend
annotations:
"consul.hashicorp.com/connect-inject": "true"
spec:
serviceAccountName: backend
containers:
- name: backend
image: nicholasjackson/fake-service:v0.22.4
ports:
- containerPort: 9090
env:
- name: "LISTEN_ADDR"
value: "0.0.0.0:9090"
- name: "NAME"
value: "backend"
- name: "MESSAGE"
value: "Response from backend"
```
</CodeBlockConfig>
1. Deploy the `backend` service to the second cluster.
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply --filename backend.yaml
```
1. In `cluster-02`, create an `ExportedServices` custom resource.
<CodeBlockConfig filename="exported-service.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ExportedServices
metadata:
name: default ## The name of the partition containing the service
spec:
services:
- name: backend ## The name of the service you want to export
consumers:
- peer: cluster-01 ## The name of the peer that receives the service
```
</CodeBlockConfig>
1. Apply the `ExportedServices` resource to the second cluster.
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply --filename exported-service.yaml
```
### Authorize services for peers
1. Create service intentions for the second cluster.
<CodeBlockConfig filename="intention.yaml">
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
name: backend-deny
spec:
destination:
name: backend
sources:
- name: "*"
action: deny
- name: frontend
action: allow
peer: cluster-01 ## The peer of the source service
```
</CodeBlockConfig>
1. Apply the intentions to the second cluster.
<CodeBlockConfig>
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT apply --filename intention.yaml
```
</CodeBlockConfig>
1. Add the `"consul.hashicorp.com/connect-inject": "true"` annotation to your service's pods before deploying the workload so that the services in `cluster-01` can dial `backend` in `cluster-02`. To dial the upstream service from an application, configure the application so that that requests are sent to the correct DNS name as specified in [Service Virtual IP Lookups](/consul/docs/discovery/dns#service-virtual-ip-lookups). In the following example, the annotation that allows the workload to join the mesh and the configuration provided to the workload that enables the workload to dial the upstream service using the correct DNS name is highlighted. [Service Virtual IP Lookups for Consul Enterprise](/consul/docs/discovery/dns#service-virtual-ip-lookups-for-consul-enterprise) details how you would similarly format a DNS name including partitions and namespaces.
<CodeBlockConfig filename="frontend.yaml" highlight="36,51">
```yaml
# Service to expose frontend
apiVersion: v1
kind: Service
metadata:
name: frontend
spec:
selector:
app: frontend
ports:
- name: http
protocol: TCP
port: 9090
targetPort: 9090
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: frontend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
labels:
app: frontend
spec:
replicas: 1
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
annotations:
"consul.hashicorp.com/connect-inject": "true"
spec:
serviceAccountName: frontend
containers:
- name: frontend
image: nicholasjackson/fake-service:v0.22.4
securityContext:
capabilities:
add: ["NET_ADMIN"]
ports:
- containerPort: 9090
env:
- name: "LISTEN_ADDR"
value: "0.0.0.0:9090"
- name: "UPSTREAM_URIS"
value: "http://backend.virtual.cluster-02.consul"
- name: "NAME"
value: "frontend"
- name: "MESSAGE"
value: "Hello World"
- name: "HTTP_CLIENT_KEEP_ALIVES"
value: "false"
```
</CodeBlockConfig>
1. Apply the service file to the first cluster.
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT apply --filename frontend.yaml
```
1. Run the following command in `frontend` and then check the output to confirm that you peered your clusters successfully.
<CodeBlockConfig highlight="31">
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT exec -it $(kubectl --context $CLUSTER1_CONTEXT get pod -l app=frontend -o name) -- curl localhost:9090
{
"name": "frontend",
"uri": "/",
"type": "HTTP",
"ip_addresses": [
"10.16.2.11"
],
"start_time": "2022-08-26T23:40:01.167199",
"end_time": "2022-08-26T23:40:01.226951",
"duration": "59.752279ms",
"body": "Hello World",
"upstream_calls": {
"http://backend.virtual.cluster-02.consul": {
"name": "backend",
"uri": "http://backend.virtual.cluster-02.consul",
"type": "HTTP",
"ip_addresses": [
"10.32.2.10"
],
"start_time": "2022-08-26T23:40:01.223503",
"end_time": "2022-08-26T23:40:01.224653",
"duration": "1.149666ms",
"headers": {
"Content-Length": "266",
"Content-Type": "text/plain; charset=utf-8",
"Date": "Fri, 26 Aug 2022 23:40:01 GMT"
},
"body": "Response from backend",
"code": 200
}
},
"code": 200
}
```
</CodeBlockConfig>
## End a peering connection
To end a peering connection, delete both the `PeeringAcceptor` and `PeeringDialer` resources.
1. Delete the `PeeringDialer` resource from the second cluster.
```shell-session
$ kubectl --context $CLUSTER2_CONTEXT delete --filename dialer.yaml
```
1. Delete the `PeeringAcceptor` resource from the first cluster.
```shell-session
$ kubectl --context $CLUSTER1_CONTEXT delete --filename acceptor.yaml
````
1. Confirm that you deleted your peering connection in `cluster-01` by querying the the `/health` HTTP endpoint. The peered services should no longer appear.
1. Exec into the server pod for the first cluster.
```shell-session
$ kubectl exec -it consul-server-0 --context $CLUSTER1_CONTEXT -- /bin/sh
```
1. If you've enabled ACLs, export an ACL token to access the `/health` HTP endpoint for services. The bootstrap token may be used if an ACL token is not already provisioned.
```shell-session
$ export CONSUL_HTTP_TOKEN=<INSERT BOOTSTRAP ACL TOKEN>
```
1. Query the the `/health` HTTP endpoint. The peered services should no longer appear.
```shell-session
$ curl "localhost:8500/v1/health/connect/backend?peer=cluster-02"
```
## Recreate or reset a peering connection
To recreate or reset the peering connection, you need to generate a new peering token from the cluster where you created the `PeeringAcceptor`.
1. In the `PeeringAcceptor` CRD, add the annotation `consul.hashicorp.com/peering-version`. If the annotation already exists, update its value to a higher version.
<CodeBlockConfig filename="acceptor.yml" highlight="6" hideClipboard>
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: PeeringAcceptor
metadata:
name: cluster-02
annotations:
consul.hashicorp.com/peering-version: "1" ## The peering version you want to set, must be in quotes
spec:
peer:
secret:
name: "peering-token"
key: "data"
backend: "kubernetes"
```
</CodeBlockConfig>
1. After updating `PeeringAcceptor`, repeat the following steps to create a peering connection:
1. [Create a peering token](#create-a-peering-token)
1. [Establish a peering connection between clusters](#establish-a-peering-connection-between-clusters)
1. [Export services between clusters](#export-services-between-clusters)
1. [Authorize services for peers](#authorize-services-for-peers)
Your peering connection is re-established with the updated token.
~> **Note:** The only way to create or set a new peering token is to manually adjust the value of the annotation `consul.hashicorp.com/peering-version`. Creating a new token causes the previous token to expire.
## Traffic management between peers
As of Consul v1.14, you can use [dynamic traffic management](/consul/docs/connect/l7-traffic) to configure your service mesh so that services automatically failover and redirect between peers.
To configure automatic service failovers and redirect, edit the `ServiceResolver` CRD so that traffic resolves to a backup service instance on a peer. The following example updates the `ServiceResolver` CRD in `cluster-01` so that Consul redirects traffic intended for the `frontend` service to a backup instance in `cluster-02` when it detects multiple connection failures to the primary instance.
<CodeBlockConfig filename="service-resolver.yaml" hideClipboard>
```yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
name: frontend
spec:
connectTimeout: 15s
failover:
'*':
targets:
- peer: 'cluster-02'
service: 'backup'
namespace: 'default'
```
</CodeBlockConfig>
Reference in New Issue View Git Blame Copy Permalink