status-im
/
consul
mirror of https://github.com/status-im/consul.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
consul/website/content/docs/k8s/annotations-and-labels.mdx

341 lines
21 KiB
Plaintext
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
layout: docs
page_title: Annotations and Labels
description: >-
Annotations and labels configure service mesh sidecar properties and injection behavior when scheduling Kubernetes clusters. Learn about the annotations and labels that enable Consul’s service mesh and secure upstream communication on k8s in this reference guide.
---
# Annotations and Labels
## Overview
Consul on Kubernetes provides a few options for customizing how connect-inject or service sync behavior should be configured.
This allows the user to configure natively configure Consul on select Kubernetes resources (i.e. pods, services).
- [Consul Service Mesh](#consul-service-mesh)
- [Annotations](#annotations)
- [Labels](#labels)
- [Service Sync](#service-sync)
- [Annotations](#annotations-1)
The noun _connect_ is used throughout this documentation to refer to the connect
subsystem that provides Consul's service mesh capabilities.
## Consul Service Mesh
### Annotations
The following Kubernetes resource annotations could be used on a pod to control connect-inject behavior:
- `consul.hashicorp.com/connect-inject` - If this is "true" then injection
is enabled. If this is "false" then injection is explicitly disabled.
The default injector behavior requires pods to opt-in to injection by
specifying this value as "true". This default can be changed in the
injector's configuration if desired.
- `consul.hashicorp.com/transparent-proxy` - If this is "true", this Pod
will run with transparent proxy enabled. This means you can use Kubernetes
DNS to access upstream services and all inbound and outbound traffic within
the pod is redirected to go through the proxy.
- `consul.hashicorp.com/transparent-proxy-overwrite-probes` - If this is "true"
and transparent proxy is enabled, the Connect injector will overwrite Kubernetes
HTTP probes to point to the Envoy proxy.
- `consul.hashicorp.com/transparent-proxy-exclude-inbound-ports` - A comma-separated
list of inbound ports to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/transparent-proxy-exclude-outbound-cidrs` - A comma-separated
list of outbound CIDRs to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/transparent-proxy-exclude-outbound-ports` - A comma-separated
list of outbound ports to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/transparent-proxy-exclude-uids` - A comma-separated
list of additional user IDs to exclude from traffic redirection when running in transparent proxy
mode.
- `consul.hashicorp.com/use-proxy-health-check` - It this is set to `true`, it configures
a readiness endpoint on Consul sidecar proxy and queries the proxy instead of the proxy's inbound port which
forwards the request to the application.
- `consul.hashicorp.com/connect-service` - For pods that accept inbound
connections, this specifies the name of the service that is being
served. This defaults to the name of the Kubernetes service associated with the pod.
If using ACLs, this must be the same name as the Pod's `ServiceAccount`.
- `consul.hashicorp.com/connect-service-port` - For pods that accept inbound
connections, this specifies the port to route inbound connections to. This
is the port that the service is listening on. The service port defaults to
the first exposed port on any container in the pod. If specified, the value
can be the _name_ of a configured port, such as "http" or it can be a direct
port value such as "8080". This is the port of the _service_, the proxy
public listener will listen on a dynamic port.
- `consul.hashicorp.com/connect-service-upstreams` - The list of upstream
services that this pod needs to connect to via the service mesh along with a static
local port to listen for those connections. When transparent proxy is enabled,
this annotation is optional. This annotation can be either _labeled_ or _unlabeled_. We recommend the labeled format because it has a more consistent syntax and can be used to reference cluster peers as upstreams.
You cannot reference auto-generated environment variables when the upstream annotation contains a dot. This is because Consul also renders the environment variables to include a dot. For example, Consul renders the variables generated for `static-server.svc:8080` as `STATIC-SERVER.SVC_CONNECT_SERVICE_HOST` and `STATIC_SERVER.SVC_CONNECT_SERVICE_PORT`, which makes the variables unusable.
- **Labeled**:
The labeled annotation format allows you to reference any service as an upstream. You can specify a Consul Enterprise namespace. You can also specify an admin partition in the same datacenter, a cluster peer, or a WAN-federated datacenter.
- Service name: Place the service name at the beginning of the annotation followed by `.svc` to specify the upstream service.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc:[port]"
```
- Peer or datacenter: Place the peer or datacenter after `svc.` followed by either `peer` or `dc` and the port number.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc.[service-peer].peer:[port]"
```
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc.[service-dc].dc:[port]"
```
- Namespace (requires Consul Enterprise): Place the namespace after `svc.` followed by `ns` and the port number.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc.[service-namespace].ns:[port]"
```
When namespaces are enabled, you must include the namespace in the annotation before specifying a cluster peer, WAN-federated datacenter, or admin partition in the same datacenter.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc.[service-namespace].ns.[service-peer].peer:[port]"
```
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc.[service-namespace].ns.[service-partition].ap:[port]"
```
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].svc.[service-namespace].ns.[service-dc].dc:[port]"
```
- **Unlabeled**:
The unlabeled annotation format allows you to reference any service not in a cluster peer as an upstream. You can specify a Consul Enterprise namespace. You can also specify an admin partition in the same datacenter or a WAN-federated datacenter. Unlike the labeled annotation, you can also reference a prepared query as an upstream.
- Service name: Place the service name at the beginning of the annotation to specify the upstream service. You also have the option to append the WAN federated datacenter where the service is deployed.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter]"
```
- Namespace: Upstream services may be running in a different namespace. Place
the upstream namespace after the service name. For additional details about configuring the injector, refer to [Consul Enterprise namespaces](#consul-enterprise-namespaces) .
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].[service-namespace]:[port]:[optional datacenter]"
```
If the namespace is not specified, the annotation defaults to the namespace of the source service.
Consul Enterprise v1.7 and older interprets the value placed in the namespace position as part of the service name.
- Admin partitions: Upstream services may be running in a different
partition. When specifying a partition, you must also specify a namespace. Place the partition name after the namespace. If you specify the name of the datacenter, it must be the local datacenter. Communicating across partitions using this method is only supported within a
datacenter. For cross partition communication across datacenters, [establish a cluster
peering connection](/consul/docs/k8s/connect/cluster-peering/usage/establish-peering) and set the upstream with a labeled annotation format.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name].[service-namespace].[service-partition]:[port]:[optional datacenter]"
```
- Prepared queries: To reference a [prepared query](/consul/api-docs/query) in an upstream annotation, prepend the annotation
with `prepared_query` and then invoke the name of the query.
```yaml
annotations:
'consul.hashicorp.com/connect-service-upstreams': 'prepared_query:[query name]:[port]'
```
- **Multiple upstreams**: Delimit multiple services or upstreams with commas. You can specify any of the unlabeled, labeled, or prepared query formats when using the supported versions for the formats.
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],[service-name]:[port]:[optional datacenter]"
```
```yaml
annotations:
"consul.hashicorp.com/connect-service-upstreams":"[service-name]:[port]:[optional datacenter],prepared_query:[query name]:[port],[service-name].svc:[port]"
```
- `consul.hashicorp.com/envoy-extra-args` - A space-separated list of [arguments](https://www.envoyproxy.io/docs/envoy/latest/operations/cli)
to be passed to the injected envoy binary.
```yaml
annotations:
consul.hashicorp.com/envoy-extra-args: '--log-level debug --disable-hot-restart'
```
- `consul.hashicorp.com/kubernetes-service` - Specifies the name of the Kubernetes service used for Consul service registration.
This is useful when multiple Kubernetes services reference the same deployment. Any service that does not match the name
specified in this annotation is ignored. When not specified no service is ignored.
```yaml
annotations:
consul.hashicorp.com/kubernetes-service: 'service-name-to-use'
```
- `consul.hashicorp.com/service-tags` - A comma separated list of tags that will
be applied to the Consul service and its sidecar.
```yaml
annotations:
consul.hashicorp.com/service-tags: foo,bar,baz
```
If you need your tag to have a comma in it you can escape the comma with `\,`. For example,
`consul.hashicorp.com/service-tags: foo\,bar\,baz` will become the single tag `foo,bar,baz`.
- `consul.hashicorp.com/service-meta-<YOUR_KEY>` - Set Consul meta key/value
pairs that will be applied to the Consul service and its sidecar.
The key will be what comes after `consul.hashicorp.com/service-meta-`, e.g.
`consul.hashicorp.com/service-meta-foo: bar` will result in `foo: bar`.
```yaml
annotations:
consul.hashicorp.com/service-meta-foo: baz
consul.hashicorp.com/service-meta-bar: baz
```
- `consul.hashicorp.com/sidecar-proxy-` - Override default resource settings for
the sidecar proxy container.
The defaults are set in Helm config via the [`connectInject.sidecarProxy.resources`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-resources) key.
- `consul.hashicorp.com/sidecar-proxy-cpu-limit` - Override the default CPU limit.
- `consul.hashicorp.com/sidecar-proxy-cpu-request` - Override the default CPU request.
- `consul.hashicorp.com/sidecar-proxy-memory-limit` - Override the default memory limit.
- `consul.hashicorp.com/sidecar-proxy-memory-request` - Override the default memory request.
- `consul.hashicorp.com/consul-envoy-proxy-concurrency` - Override the default envoy worker thread count. This should be set low for sidecar
use cases and can be raised for edge proxies like gateways.
- `consul.hashicorp.com/consul-sidecar-` - Override default resource settings for
the `consul-sidecar` container.
The defaults are set in Helm config via the [`global.consulSidecarContainer.resources`](/consul/docs/k8s/helm#v-global-consulsidecarcontainer) key.
- `consul.hashicorp.com/consul-sidecar-cpu-limit` - Override the default CPU limit.
- `consul.hashicorp.com/consul-sidecar-cpu-request` - Override the default CPU request.
- `consul.hashicorp.com/consul-sidecar-memory-limit` - Override the default memory limit.
- `consul.hashicorp.com/consul-sidecar-memory-request` - Override the default memory request.
- `consul.hashicorp.com/enable-sidecar-proxy-lifecycle` - Override the default Helm value [`connectInject.sidecarProxy.lifecycle.defaultEnabled`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-lifecycle-defaultenabled)
- `consul.hashicorp.com/enable-sidecar-proxy-shutdown-drain-listeners` - Override the default Helm value [`connectInject.sidecarProxy.lifecycle.defaultEnableShutdownDrainListeners`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-lifecycle-defaultenableshutdowndrainlisteners)
- `consul.hashicorp.com/sidecar-proxy-lifecycle-shutdown-grace-period-seconds` - Override the default Helm value [`connectInject.sidecarProxy.lifecycle.defaultShutdownGracePeriodSeconds`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-lifecycle-defaultshutdowngraceperiodseconds)
- `consul.hashicorp.com/sidecar-proxy-lifecycle-graceful-port` - Override the default Helm value [`connectInject.sidecarProxy.lifecycle.defaultGracefulPort`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-lifecycle-defaultgracefulport)
- `consul.hashicorp.com/sidecar-proxy-lifecycle-graceful-shutdown-path` - Override the default Helm value [`connectInject.sidecarProxy.lifecycle.defaultGracefulShutdownPath`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-lifecycle-defaultgracefulshutdownpath)
- `consul.hashicorp.com/sidecar-proxy-startup-failure-seconds` - Override the default Helm value [`connectInject.sidecarProxy.defaultStartupFailureSeconds`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-defaultstartupfailureseconds)
- `consul.hashicorp.com/sidecar-proxy-liveness-failure-seconds` - Override the default Helm value [`connectInject.sidecarProxy.defaultLivenessFailureSeconds`](/consul/docs/k8s/helm#v-connectinject-sidecarproxy-defaultlivenessfailureseconds)
- `consul.hashicorp.com/enable-metrics` - Override the default Helm value [`connectInject.metrics.defaultEnabled`](/consul/docs/k8s/helm#v-connectinject-metrics-defaultenabled).
- `consul.hashicorp.com/enable-metrics-merging` - Override the default Helm value [`connectInject.metrics.defaultEnableMerging`](/consul/docs/k8s/helm#v-connectinject-metrics-defaultenablemerging).
- `consul.hashicorp.com/merged-metrics-port` - Override the default Helm value [`connectInject.metrics.defaultMergedMetricsPort`](/consul/docs/k8s/helm#v-connectinject-metrics-defaultmergedmetricsport).
- `consul.hashicorp.com/prometheus-scrape-port` - Override the default Helm value [`connectInject.metrics.defaultPrometheusScrapePort`](/consul/docs/k8s/helm#v-connectinject-metrics-defaultprometheusscrapeport).
- `consul.hashicorp.com/prometheus-scrape-path` - Override the default Helm value [`connectInject.metrics.defaultPrometheusScrapePath`](/consul/docs/k8s/helm#v-connectinject-metrics-defaultprometheusscrapepath).
- `consul.hashicorp.com/prometheus-ca-file` - Local filesystem path to a CA file for Envoy to use
when serving TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
is set in proxy config.
- `consul.hashicorp.com/prometheus-ca-path` - Local filesystem path to a directory of CA certificates
for Envoy to use when serving TLS on the Prometheus metrics endpoint. Only applicable when
`envoy_prometheus_bind_addr` is set in proxy config.
- `consul.hashicorp.com/prometheus-cert-file` - Local filesystem path to a certificate file for Envoy to use
when serving TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
is set in proxy config.
- `consul.hashicorp.com/prometheus-key-file` - Local filesystem path to a private key file for Envoy to use
when serving TLS on the Prometheus metrics endpoint. Only applicable when `envoy_prometheus_bind_addr`
is set in proxy config.
- `consul.hashicorp.com/service-metrics-port` - Set the port where the mesh service exposes metrics.
- `consul.hashicorp.com/service-metrics-path` - Set the path where the mesh service exposes metrics.
- `consul.hashicorp.com/connect-inject-mount-volume` - Comma separated list of container names to mount the connect-inject volume into. The volume will be mounted at `/consul/connect-inject`. The connect-inject volume contains Consul internals data needed by the other sidecar containers, for example the `consul` binary, and the Pod's Consul ACL token. This data can be valuable for advanced use-cases, such as making requests to the Consul API from within application containers.
- `consul.hashicorp.com/consul-sidecar-user-volume` - JSON objects as specified by the [Volume pod spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#volume-v1-core), that define volumes to add to the Envoy sidecar.
```yaml
annotations:
"consul.hashicorp.com/consul-sidecar-user-volume": "[{\"name\": \"secrets-data\", \"hostPath\": "[{\"path\": \"/mnt/secrets-path\"}]"}]"
```
- `consul.hashicorp.com/consul-sidecar-user-volume-mount` - JSON objects as specified by the [Volume mount pod spec](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#volumemount-v1-core), that define volumeMounts to add to the Envoy sidecar.
```yaml
annotations:
"consul.hashicorp.com/consul-sidecar-user-volume-mount": "[{\"name\": \"secrets-store-mount\", \"mountPath\": \"/mnt/secrets-store\"}]"
```
- `consul.hashicorp.com/proxy-config-map` - JSON object specifying [Proxy Config Options](/consul/docs/connect/proxies/envoy#proxy-config-options). The proxy config map provides a low-level interface for setting configuration fields on a per-proxy basis during service registration. This configuration field is intended to be used in situations where a field does not exist in [service defaults configuration entries](/consul/docs/connect/config-entries/service-defaults) and another annotation does not provide explicit access to one of the Envoy configuration options.
```yaml
annotations:
"consul.hashicorp.com/proxy-config-map": "{ \"xds_fetch_timeout_ms\": 30000 }"
```
### Labels
Resource labels could be used on a Kubernetes service to control connect-inject behavior.
- `consul.hashicorp.com/service-ignore` - This label can be set on a Kubernetes Service.
If set to "true", the service will not be used to register a Consul endpoint. This can be
useful in cases where 2 or more services point to the same deployment. Consul cannot register
multiple endpoints to the same deployment. This label can be used to tell the endpoint
registration to ignore all services except for the one which should be used for routing requests
using Consul.
## Service Sync
### Annotations
The following Kubernetes resource annotations could be used on a pod to [Service Sync](https://developer.hashicorp.com/consul/docs/k8s/service-sync) behavior:
- `consul.hashicorp.com/service-sync`: If this is set to `true`, then the Kubernetes service is explicitly configured to be synced to Consul.
```yaml
annotations:
'consul.hashicorp.com/service-sync': 'true'
```
- `consul.hashicorp.com/service-port`: Configures the port to register to the Consul Catalog for the Kubernetes service. The annotation value may be a name of a port (recommended) or an exact port value. Refer to [service ports](https://developer.hashicorp.com/consul/docs/k8s/service-sync#service-ports) for more information.
```yaml
annotations:
'consul.hashicorp.com/service-port': 'http'
```
- `consul.hashicorp.com/service-tags`: A comma separated list of strings (without whitespace) to use for registering tags to the service registered to Consul. These custom tags automatically include the `k8s` tag which can't be disabled.
```yaml
annotations:
'consul.hashicorp.com/service-tags': 'primary,foo'
```
- `consul.hashicorp.com/service-meta-KEY`: A map for specifying service metadata for Consul services. The "KEY" below can be set to any key. This allows you to set multiple meta values.
```yaml
annotations:
'consul.hashicorp.com/service-meta-KEY': 'value'
```
- `consul.hashicorp.com/service-weight:` - Configures ability to support weighted loadbalancing by service annotation for Catalog Sync. The integer provided will be applied as a weight for the `passing` state for the health of the service. Refer to [weights](/consul/docs/services/configuration/services-configuration-reference#weights) in service configuration for more information on how this is leveraged for services in the Consul catalog.
```yaml
annotations:
consul.hashicorp.com/service-weight: 10
```
Reference in New Issue View Git Blame Copy Permalink