mirror of https://github.com/status-im/consul.git
461 lines
17 KiB
Plaintext
461 lines
17 KiB
Plaintext
---
|
||
layout: docs
|
||
page_title: Service Sync - Kubernetes
|
||
description: >-
|
||
The services in Kubernetes and Consul can be automatically synced so that
|
||
Kubernetes services are available to Consul agents and services in Consul can
|
||
be available as first-class Kubernetes services.
|
||
---
|
||
|
||
# Syncing Kubernetes and Consul Services
|
||
|
||
The services in Kubernetes and Consul can be automatically synced so that Kubernetes
|
||
services are available to Consul agents and services in Consul can be available
|
||
as first-class Kubernetes services. This functionality is provided by the
|
||
[consul-k8s project](https://github.com/hashicorp/consul-k8s) and can be
|
||
automatically installed and configured using the
|
||
[Consul Helm chart](/docs/k8s/installation/install).
|
||
|
||
|
||
|
||
**Why sync Kubernetes services to Consul?** Kubernetes services synced to the
|
||
Consul catalog enable Kubernetes services to be accessed by any node that
|
||
is part of the Consul cluster, including other distinct Kubernetes clusters.
|
||
For non-Kubernetes nodes, they can access services using the standard
|
||
[Consul DNS](/docs/discovery/dns) or HTTP API.
|
||
|
||
**Why sync Consul services to Kubernetes?** Syncing Consul services to
|
||
Kubernetes services enables non-Kubernetes services (such as external to
|
||
the cluster) to be accessed in a native Kubernetes way: using kube-dns,
|
||
environment variables, etc. This makes it very easy to automate external
|
||
service discovery, including hosted services like databases.
|
||
|
||
## Installation and Configuration
|
||
|
||
The service sync is done using an external long-running process in the
|
||
[consul-k8s project](https://github.com/hashicorp/consul-k8s). This process
|
||
can run either in or out of a Kubernetes cluster. However, running this within
|
||
the Kubernetes cluster is generally easier since it is automated using the
|
||
[Helm chart](/docs/k8s/helm).
|
||
|
||
The Consul server cluster can run either in or out of a Kubernetes cluster.
|
||
The Consul server cluster does not need to be running on the same machine
|
||
or same platform as the sync process. The sync process needs to be configured
|
||
with the address to a Consul agent as well as any additional access
|
||
information such as ACL tokens.
|
||
|
||
To install the sync process, enable the catalog sync feature using
|
||
[Helm values](/docs/k8s/helm#configuration-values) and
|
||
upgrade the installation using `helm upgrade` for existing installs or
|
||
`helm install` for a fresh install.
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
```
|
||
|
||
This will enable services to sync _in both directions_. You can also choose
|
||
to only sync Kubernetes services to Consul or vice versa by disabling a direction.
|
||
|
||
To only enable syncing Consul services to Kubernetes use the config:
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
toConsul: false
|
||
toK8S: true
|
||
```
|
||
|
||
To only enable syncing Kubernetes services to Consul use:
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
toConsul: true
|
||
toK8S: false
|
||
```
|
||
|
||
See the [Helm configuration](/docs/k8s/helm#v-synccatalog)
|
||
for more information.
|
||
|
||
### Authentication
|
||
|
||
The sync process must authenticate to both Kubernetes and Consul to read
|
||
and write services.
|
||
|
||
If running `consul-k8s` using the Helm chart then this authentication is handled for you.
|
||
|
||
If running `consul-k8s` outside of Kubernetes, a valid kubeconfig file must be provided with cluster
|
||
and authentication information. The sync process will look into the default locations
|
||
for both in-cluster and out-of-cluster authentication. If `kubectl` works,
|
||
then the sync program should work.
|
||
|
||
For Consul, if ACLs are configured on the cluster, a Consul
|
||
[ACL token](https://learn.hashicorp.com/tutorials/consul/access-control-setup-production)
|
||
will need to be provided. Review the [ACL rules](/docs/security/acl/acl-rules)
|
||
when creating this token so that it only allows the necessary privileges. The catalog
|
||
sync process accepts this token by using the [`CONSUL_HTTP_TOKEN`](/commands#consul_http_token)
|
||
environment variable. This token should be set as a
|
||
[Kubernetes secret](https://kubernetes.io/docs/concepts/configuration/secret/#creating-your-own-secrets)
|
||
and referenced in the Helm chart.
|
||
|
||
## Kubernetes to Consul
|
||
|
||
This sync registers Kubernetes services to the Consul catalog automatically.
|
||
|
||
This enables discovery and connection to Kubernetes services using native
|
||
Consul service discovery such as DNS or HTTP. This is particularly useful for
|
||
non-Kubernetes nodes. This also causes all discoverable services to be part of
|
||
a central service catalog in Consul for further syncing into alternate
|
||
Kubernetes clusters or other platforms.
|
||
|
||
Each synced service will be registered onto a Consul node called `k8s-sync`. This
|
||
is not a real node because there is no Consul client registering and monitoring
|
||
the services. Instead, the catalog sync process is monitoring Kubernetes
|
||
and syncing the services to Consul.
|
||
|
||
### Kubernetes Service Types
|
||
|
||
Not all Kubernetes services are externally accessible. The sync program by
|
||
default will only sync services of the following types or configurations.
|
||
If a service type is not listed below, then the sync program will ignore that
|
||
service type.
|
||
|
||
#### NodePort
|
||
|
||
[NodePort services](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport)
|
||
register a static port that every node in the K8S cluster listens on.
|
||
|
||
For NodePort services, a Consul service instance will be created for each
|
||
node that has the representative pod running. While Kubernetes configures
|
||
a static port on all nodes in the cluster, this limits the number of service
|
||
instances to be equal to the nodes running the target pods.
|
||
By default it will use the external IP of the node but this can be configured via
|
||
the [`nodePortSyncType` helm option](/docs/k8s/helm#v-synccatalog-nodeportsynctype).
|
||
|
||
The service instance's port will be set to the _first_ defined node port of the service unless
|
||
set specifically via the `consul.hashicorp.com/service-port` annotation (see [Service Ports](/docs/k8s/service-sync#service-ports)).
|
||
|
||
#### LoadBalancer
|
||
|
||
For LoadBalancer services, a single service instance will be registered with
|
||
the external IP of the created load balancer. Because this is already a load
|
||
balancer, only one service instance will be registered with Consul rather
|
||
than registering each individual pod endpoint.
|
||
|
||
The service instance's port will be set to the _first_ defined port of the
|
||
service unless set specifically via the `consul.hashicorp.com/service-port` annotation (see [Service Ports](/docs/k8s/service-sync#service-ports)).
|
||
|
||
#### External IPs
|
||
|
||
Any service type may specify an
|
||
"[external IP](https://kubernetes.io/docs/concepts/services-networking/service/#external-ips)"
|
||
configuration. The external IP must be configured by some other system, but
|
||
any service discovery will resolve to this set of IP addresses rather than a
|
||
virtual IP.
|
||
|
||
If an external IP list is present, a service instance in Consul will be created
|
||
for each external IP. It is assumed that if an external IP is present that it
|
||
is routable and configured by some other system.
|
||
|
||
The service instance's port will be set to the _first_ defined port of the
|
||
service unless set specifically via the `consul.hashicorp.com/service-port` annotation (see [Service Ports](/docs/k8s/service-sync#service-ports)).
|
||
|
||
#### ClusterIP
|
||
|
||
ClusterIP services are synced by default as of `consul-k8s` version 0.3.0.
|
||
Each pod that is an endpoint for the service will be synced as a Consul service
|
||
instance with its IP set to the pod IP and its port set to the `targetPort`.
|
||
|
||
The service instance's port can be overridden via the `consul.hashicorp.com/service-port` annotation (see [Service Ports](/docs/k8s/service-sync#service-ports)).
|
||
|
||
-> In Kubernetes clusters where pod IPs are not accessible outside the cluster,
|
||
the services registered in Consul may not be routable. To
|
||
skip syncing ClusterIP services, set [`syncClusterIPServices`](/docs/k8s/helm#v-synccatalog-syncclusteripservices)
|
||
to `false` in the Helm chart values file.
|
||
|
||
### Sync Enable/Disable
|
||
|
||
By default, all valid service types (as explained above) are synced from every Kubernetes
|
||
namespace (except for `kube-system` and `kube-public`).
|
||
If you wish to only sync specific services via annotation, set the default to `false`:
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
default: false
|
||
```
|
||
|
||
And explicitly enable syncing specific services via the `consul.hashicorp.com/service-sync` annotation:
|
||
|
||
```yaml
|
||
kind: Service
|
||
apiVersion: v1
|
||
metadata:
|
||
name: my-service
|
||
annotations:
|
||
'consul.hashicorp.com/service-sync': 'true'
|
||
```
|
||
|
||
-> **NOTE:** If the annotation is set to `false` when the default sync is `true`, the service will **not** be synced.
|
||
|
||
You can allow or deny syncing from specific Kubernetes namespaces by setting the
|
||
`k8sAllowNamespaces` and `k8sDenyNamespaces` keys:
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
default: true
|
||
k8sAllowNamespaces: ['*']
|
||
k8sDenyNamespaces: ['kube-system', 'kube-public']
|
||
```
|
||
|
||
In the default configuration (shown above), services from all namespaces except for
|
||
`kube-system` and `kube-public` will be synced.
|
||
|
||
If you wish to only sync from specific namespaces, you can list only those
|
||
namespaces in the `k8sAllowNamespaces` key:
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
default: true
|
||
k8sAllowNamespaces: ['my-ns-1', 'my-ns-2']
|
||
k8sDenyNamespaces: []
|
||
```
|
||
|
||
If you wish to sync from every namespace _except_ specific namespaces, you can
|
||
use `*` in the allow list and then specify the non-syncing namespaces in the deny list:
|
||
|
||
```yaml
|
||
syncCatalog:
|
||
enabled: true
|
||
default: true
|
||
k8sAllowNamespaces: ['*']
|
||
k8sDenyNamespaces: ['no-sync-ns-1', 'no-sync-ns-2']
|
||
```
|
||
|
||
-> **NOTE:** The deny list takes precedence over the allow list. If a namespace
|
||
is listed in both lists, it will **not** be synced.
|
||
|
||
### Service Name
|
||
|
||
When a Kubernetes service is synced to Consul, the name of the service in Consul
|
||
by default will be the value of the "name" metadata on that Kubernetes service.
|
||
This makes it so that service sync works with zero configuration changes.
|
||
This can be overridden using an annotation to specify the Consul service name:
|
||
|
||
```yaml
|
||
kind: Service
|
||
apiVersion: v1
|
||
metadata:
|
||
name: my-service
|
||
annotations:
|
||
'consul.hashicorp.com/service-name': my-consul-service
|
||
```
|
||
|
||
**If a conflicting service name exists in Consul,** the sync program will
|
||
register additional instances to that same service. Therefore, services inside
|
||
and outside of Kubernetes should have different names unless you want either
|
||
side to potentially connect. This default behavior also enables gracefully
|
||
transitioning a service from outside of K8S to inside, and vice versa.
|
||
|
||
### Service Ports
|
||
|
||
When syncing the Kubernetes service to Consul, the Consul service port will be
|
||
the first defined port in the service. Additionally, all ports will be
|
||
registered in the service instance metadata with the key "port-X" where X is
|
||
the name of the port and the value is the externally accessible port.
|
||
|
||
The default service port can be overridden using an annotation:
|
||
|
||
```yaml
|
||
kind: Service
|
||
apiVersion: v1
|
||
metadata:
|
||
name: my-service
|
||
annotations:
|
||
'consul.hashicorp.com/service-port': 'http'
|
||
```
|
||
|
||
The annotation value may be a name of a port (recommended) or an exact port value.
|
||
|
||
### Service Tags
|
||
|
||
A service registered in Consul from Kubernetes will always have the tag "k8s" added
|
||
to it. Additional tags can be specified with a comma-separated annotation value
|
||
as shown below. This will also automatically include the "k8s" tag which can't
|
||
be disabled. The values should be specified comma-separated without any
|
||
additional whitespace.
|
||
|
||
```yaml
|
||
kind: Service
|
||
apiVersion: v1
|
||
metadata:
|
||
name: my-service
|
||
annotations:
|
||
'consul.hashicorp.com/service-tags': 'primary,foo'
|
||
```
|
||
|
||
### Service Meta
|
||
|
||
A service registered in Consul from Kubernetes will set the `external-source` key to
|
||
"kubernetes". This can be used by API consumers, the UI, CLI, etc. to filter
|
||
service instances that are set in k8s. The Consul UI (in Consul 1.2.3 and later)
|
||
will read this value to show a Kubernetes icon next to all externally
|
||
registered services from Kubernetes.
|
||
|
||
Additional metadata can be specified using annotations. The "KEY" below can be
|
||
set to any key. This allows setting multiple meta values:
|
||
|
||
```yaml
|
||
kind: Service
|
||
apiVersion: v1
|
||
metadata:
|
||
name: my-service
|
||
annotations:
|
||
'consul.hashicorp.com/service-meta-KEY': 'value'
|
||
```
|
||
|
||
### Consul Enterprise Namespaces
|
||
|
||
Consul Enterprise supports Consul namespaces. These can be used when syncing
|
||
from Kubernetes to Consul (although not vice-versa).
|
||
|
||
There are three options available:
|
||
|
||
1. **Single Destination Namespace** – Sync all Kubernetes services, regardless of namespace,
|
||
into the same Consul namespace.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
syncCatalog:
|
||
enabled: true
|
||
consulNamespaces:
|
||
consulDestinationNamespace: 'my-consul-ns'
|
||
```
|
||
|
||
1. **Mirror Namespaces** - Each Kubernetes service will be synced to a Consul namespace with the same
|
||
name as its Kubernetes namespace.
|
||
|
||
For example, service `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace `ns-1`.
|
||
If a mirrored namespace does not exist in Consul, it will be created.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
syncCatalog:
|
||
enabled: true
|
||
consulNamespaces:
|
||
mirroringK8S: true
|
||
|
||
addK8SNamespaceSuffix: false
|
||
```
|
||
|
||
1. **Mirror Namespaces With Prefix** - Each Kubernetes service will be synced to a Consul namespace
|
||
with the same name as its Kubernetes namespace **with a prefix**. For example, given a prefix
|
||
`k8s-`, service `foo` in Kubernetes namespace `ns-1` will be synced to the Consul namespace
|
||
`k8s-ns-1`.
|
||
|
||
This can be configured with:
|
||
|
||
```yaml
|
||
global:
|
||
enableConsulNamespaces: true
|
||
|
||
syncCatalog:
|
||
enabled: true
|
||
consulNamespaces:
|
||
mirroringK8S: true
|
||
mirroringK8SPrefix: 'k8s-'
|
||
|
||
addK8SNamespaceSuffix: false
|
||
```
|
||
|
||
-> Note that in both mirroring examples we're setting `addK8SNamespaceSuffix: false`. If set to `true`
|
||
(the default), the Kubernetes namespace will be added as a suffix to each
|
||
Consul service name. For example Kubernetes service `foo` in namespace `k8s-ns`
|
||
would be registered into Consul with the name `foo-k8s-ns`.
|
||
This is useful when syncing from multiple Kubernetes namespaces to
|
||
a single consul namespace but is likely something you'll want turned off
|
||
when mirroring namespaces since services won't overlap with services from
|
||
other namespaces.
|
||
|
||
## Consul to Kubernetes
|
||
|
||
This syncs Consul services into first-class Kubernetes services.
|
||
The sync service will create an [`ExternalName`](https://kubernetes.io/docs/concepts/services-networking/service/#externalname)
|
||
`Service` for each Consul service. The "external name" will be
|
||
the Consul DNS name.
|
||
|
||
For example, given a Consul service `foo`, a Kubernetes Service will be created
|
||
as follows:
|
||
|
||
```yaml
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: foo
|
||
...
|
||
spec:
|
||
externalName: foo.service.consul
|
||
type: ExternalName
|
||
```
|
||
|
||
With Consul To Kubernetes syncing enabled, DNS requests of the form `<consul-service-name>`
|
||
will be serviced by Consul DNS. From a different Kubernetes namespace than where Consul
|
||
is deployed, the DNS request would need to be `<consul-service-name>.<consul-namespace>`.
|
||
|
||
-> **Note:** Consul to Kubernetes syncing **isn't required** if you've enabled [Consul DNS on Kubernetes](/docs/k8s/dns)
|
||
_and_ all you need to do is address services in the form `<consul-service-name>.service.consul`, i.e. you don't need Kubernetes `Service` objects created.
|
||
|
||
~> **Requires Consul DNS via CoreDNS in Kubernetes:** This feature requires that
|
||
[Consul DNS](/docs/k8s/dns) is configured within Kubernetes.
|
||
Additionally, **[CoreDNS](https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#config-coredns)
|
||
is required (instead of kube-dns)** to resolve an
|
||
issue with resolving `externalName` services pointing to custom domains.
|
||
|
||
### Sync Enable/Disable
|
||
|
||
All Consul services visible to the sync process based on its given ACL token
|
||
will be synced to Kubernetes.
|
||
|
||
There is no way to change this behavior per service. For the opposite sync
|
||
direction (Kubernetes to Consul), you can use Kubernetes annotations to disable
|
||
a sync per service. This is not currently possible for Consul to Kubernetes
|
||
sync and the ACL token must be used to limit what services are synced.
|
||
|
||
In the future, we hope to support per-service configuration.
|
||
|
||
### Service Name
|
||
|
||
When a Consul service is synced to Kubernetes, the name of the Kubernetes
|
||
service will exactly match the name of the Consul service.
|
||
|
||
To change this default exact match behavior, it is possible to specify a
|
||
prefix to be added to service names within Kubernetes by using the
|
||
`-k8s-service-prefix` flag. This can also be specified in the Helm
|
||
configuration.
|
||
|
||
**If a conflicting service is found,** the service will not be synced. This
|
||
does not match the Kubernetes to Consul behavior, but given the current
|
||
implementation we must do this because Kubernetes can't mix both CNAME and
|
||
Endpoint-based services.
|
||
|
||
### Kubernetes Service Labels and Annotations
|
||
|
||
Any Consul services synced to Kubernetes will be labeled and annotated.
|
||
An annotation `consul.hashicorp.com/synced` will be set to "true" to note
|
||
that this is a synced service from Consul.
|
||
|
||
Additionally, a label `consul=true` will be specified so that label selectors
|
||
can be used with `kubectl` and other tooling to easily filter all Consul-synced
|
||
services.
|