Envoy L7 config docs (#5809)

* WIP

* Document all the new Envoy L7 configs

* Apply suggestions from code review

Co-Authored-By: banks <banks@banksco.de>

* Rewrite dynamic config and add in TODO links
This commit is contained in:
Paul Banks 2019-05-08 22:03:53 +01:00 committed by GitHub
parent c4973a63f3
commit 5a67ee72d0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -14,257 +14,330 @@ optionally exposing a gRPC service on the local agent that serves [Envoy's xDS
configuration configuration
API](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md). API](https://github.com/envoyproxy/data-plane-api/blob/master/XDS_PROTOCOL.md).
Currently Consul only supports TCP proxying between services, however HTTP and Consul can configure Envoy sidecars to proxy http/1.1, http2 or gRPC traffic at
gRPC features are planned for the near future along with first class ways to L7 or any other tcp-based protocol at L4. Prior to Consul 1.5.0 Envoy proxies
configure them in Consul. could only proxy tcp at L4.
As an interim solution, [custom Envoy configuration](#custom-configuration) can Currently configuration of additional L7 features is limited, however we have
be specified in [proxy service definition](/docs/connect/proxies.html) allowing plans to support a wider range of features in the next major release
more powerful features of Envoy to be used. cycle.
As an interim solution, you can add [custom Envoy configuration](#custom-configuration)
in the [proxy service definition](/docs/connect/proxies.html) allowing
you to use the more powerful features of Envoy.
## Supported Versions ## Supported Versions
Consul's Envoy support was added in version 1.3.0. It has been tested against Consul's Envoy support was added in version 1.3.0. The following table shows
Envoy 1.7.1 and 1.8.0. compatible Envoy versions.
| Consul Version | Compatible Envoy Versions |
|---|---|
| 1.5.x and higher | 1.9.1, 1.8.0† |
| 1.3.x, 1.4.x | 1.9.1, 1.8.0†, 1.7.0† |
!> **Security Note:** Envoy versions lower than 1.9.1 are vulnerable to
[CVE-2019-9900](https://github.com/envoyproxy/envoy/issues/6434) and
[CVE-2019-9901](https://github.com/envoyproxy/envoy/issues/6435). Both are
related to HTTP request parsing and so only affect Consul Connect users if they
have configured HTTP routing rules via the ["escape
hatch"](#custom-configuration). Still, we recommend that you use Envoy 1.9.1 where
possible.
## Getting Started ## Getting Started
To get started with Envoy and see a working example you can follow the [Using To get started with Envoy and see a working example you can follow the [Using
Envoy with Connect](/docs/guides/connect-envoy.html) guide. Envoy with Connect](/docs/guides/connect-envoy.html) guide.
## Limitations ## Configuration
The following list limitations of the Envoy integration as released in 1.3.0. Envoy proxies require two types of configuration: an initial _bootstrap
All of these are planned to be lifted in the near future. configuration_ and dynamic configuration that is discovered from a "management
server", in this case Consul.
* Default Envoy configuration only supports Layer 4 (TCP) proxying. More The bootstrap configuration at a minimum needs to configure the proxy with an
[advanced listener configuration](#advanced-listener-configuration) is identity (node id) and the location of it's local Consul agent from which it
possible but experimental and requires deep Envoy knowledge. First class discovers all of it's dynamic configuration. See [Bootstrap
workflows for configuring Layer 7 features across the cluster are planned for Configuration](#bootstrap-configuration) for more details.
the near future.
* There is currently no way to override the configuration of upstream clusters The dynamic configuration Consul Connect provides to each Envoy instance includes:
which makes it impossible to configure Envoy features like circuit breakers,
load balancing policy, custom protocol settings etc. This will be fixed in a - TLS certificates and keys to enable mutual authentication and keep certificates
near-future release first with an "escape hatch" similar to the one for rotating.
listeners below, then later with first-class support. - Service-discovery results for upstreams to enable each sidecar proxy to load-balance
* The configuration delivered to Envoy is suitable for a sidecar proxy outgoing connections.
currently. Later we plan to support more flexibility to be able to configure - L7 configuration including timeouts and protocol-specific options.
Envoy as an edge router or gateway and similar.
* There is currently no way to disable the public listener and have a "client For more information on the parts of the Envoy proxy runtime configuration
only" sidecar for services that don't expose Connect-enabled service but want that are currently controllable via Consul Connect see [Dynamic
to consume others. This will be fixed in a near-future release. Configuration](#dynamic-configuration).
* Once authorized, a persistent TCP connection will not be closed if the
intentions change to deny access. This is currently a limitation of how TCP We plan to enable more and more of Envoy's features through
proxy and network authz filter work in Envoy. All new connections will be Connect's first-class configuration over time, however some advanced users will
denied though and destination services can limit exposure by closing inbound need additional control to configure Envoy in specific ways. To enable this, we
connections periodically or by a rolling restart of the destination service provide several ["escape hatch"](#advanced-configuration) options that allow
as an emergency measure. users to provide low-level raw Envoy config syntax for some sub-components in each
Envoy instance. This allows operators to have full control over and
responsibility for correctly configuring Envoy and ensuring version support etc.
## Bootstrap Configuration ## Bootstrap Configuration
Envoy requires an initial bootstrap configuration that directs it to the local Envoy requires an initial bootstrap configuration file. The easiest way to
agent for further configuration discovery. create this is using the [`consul connect envoy`
To assist in generating this, Consul 1.3.0 adds a [`consul connect envoy`
command](/docs/commands/connect/envoy.html). The command can either output the command](/docs/commands/connect/envoy.html). The command can either output the
bootstrap configuration directly or can generate it and then `exec` the Envoy bootstrap configuration directly to stdout or can generate it and then `exec`
binary as a convenience wrapper. the Envoy binary as a convenience wrapper.
Some Envoy configuration options like metrics and tracing sinks can only be Because some Envoy configuration options like metrics and tracing sinks can only be
specified via the bootstrap config currently and so a custom bootstrap must be specified via the bootstrap configuration, Connect as of Consul 1.5.0 adds
used. In order to work with Connect it's necessary to start with the following the ability to control some parts of the bootstrap config via proxy
basic template and add additional configuration as needed. configuration options.
```yaml Users can add the following configuration items to the [global `proxy-defaults`
admin: configuration entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) or override them directly in the `proxy.config` field
# access_log_path and address are required by Envoy, Consul doesn't care what of a [proxy service
# they are set to though and never accesses the admin API. definition](/docs/connect/proxies.html#proxy-service-definitions) or
node: [`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block.
# cluter is required by Envoy but Consul doesn't use it
cluster: "<cluster_name"
# id must be the ID (not name if they differ) of the proxy service
# registration in Consul
id: "<proxy_service_id>"
static_resources:
clusters:
# local_agent is the "cluster" used to make further discovery requests for
# config and should point to the gRPC port of the local Consul agent instance.
- name: local_agent
connect_timeout: 1s
type: STATIC
# tls_context is needed if and only if Consul agent TLS is configured
tls_context:
common_tls_context:
validation_context:
trusted_ca:
filename: "<path to CA cert file Consul is using>"
http2_protocol_options: {}
hosts:
- socket_address:
address: "<agent's local IP address, usually 127.0.0.1>"
port_value: "<agent's grpc port, default 8502>"
dynamic_resources:
lds_config:
ads: {}
cds_config:
ads: {}
ads_config:
api_type: GRPC
grpc_services:
initial_metadata:
- key: "x-consul-token"
token: "<Consul ACL token with service:write on the target service>"
envoy_grpc:
cluster_name: local_agent
```
This configures a "cluster" pointing to the local Consul agent and sets that as - `envoy_statsd_url` - A URL in the form `udp://ip:port` identifying a UDP
the target for discovering all types of dynamic resources. StatsD listener that Envoy should deliver metrics to. For example, this may be
`udp://127.0.0.1:8125` if every host has a local StatsD listener. In this case
users can configure this property once in the [global `proxy-defaults`
configuration entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) for convenience. Currently, TCP is not supported.
~> **Security Note**: The bootstrap configuration must contain the Consul ACL ~> **Note:** currently the url **must use an ip address** not a dns name due
token authorizing the proxy to identify as the target service. As such it should to the way Envoy is setup for StatsD.
be treated as a secret value and handled with care - an attacker with access to
one is able to obtain Connect TLS certificates for the target service and so
access anything that service is authorized to connect to.
## Advanced Listener Configuration Users can also specify the whole parameter in the form `$ENV_VAR_NAME`, which
will cause the `consul connect envoy` command to resolve the actual URL from
the named environment variable when it runs. This, for example, allows each
pod in a Kubernetes cluster to learn of a pod-specific IP address for StatsD
when the Envoy instance is bootstrapped while still allowing global
configuration of all proxies to use StatsD in the [global `proxy-defaults`
configuration entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults). The env variable must contain a full valid URL
value as specified above and nothing else. It is not currently possible to use
environment variables as only part of the URL.
Consul 1.3.0 includes initial Envoy support which includes automatic Layer 4 - `envoy_dogstatsd_url` - The same as `envoy_statsd_url` with the following
(TCP) proxying over mTLS, and authorization. Near future versions of Consul will differences in behavior:
bring Layer 7 features like HTTP-path-based routing, retries, tracing and more. - Envoy will use dogstatsd tags instead of statsd dot-separated metric names.
- As well as `udp://`, a `unix://` URL may be specified if your agent can
listen on a unix socket (e.g. the dogstatsd agent).
-> **Advanced Topic!** This section covers an optional way of taking almost - `envoy_prometheus_bind_addr` - Specifies that the proxy should expose a Prometheus
complete control of Envoy's listener configuration which is provided as a way to metrics endpoint to the _public_ network. It must be supplied in the form
experiment with advanced integrations ahead of full layer 7 feature support. `ip:port` and port and the ip/port combination must be free within the network
While we don't plan to remove the ability to do this in the future, it should be namespace the proxy runs. Typically the IP would be `0.0.0.0` to bind to all
considered experimental and requires in-depth knowledge of Envoy's configuration available interfaces or a pod IP address.
format.
For advanced users there is an "escape hatch" available in 1.3.0. The -> **Note:** Envoy versions prior to 1.10 do not export timing histograms
`proxy.config` map in the [proxy service using the internal Prometheus endpoint. Consul 1.5.0 [doesn't yet support
definition](/docs/connect/proxies.html#proxy-service-definitions) may contain a Envoy 1.10](#supported-versions) although support will soon be added.
special key called `envoy_public_listener_json`. If this is set, it's value must
be a string containing the serialized proto3 JSON encoding of a complete [envoy
listener
config](https://www.envoyproxy.io/docs/envoy/v1.8.0/api-v2/api/v2/lds.proto).
Each upstream listener may also be customized in the same way by adding a
`envoy_listener_json` key to the `config` map of [the upstream
definition](/docs/connect/proxies.html#upstream-configuration-reference).
The JSON supplied may describe a protobuf `types.Any` message with `@type` set - `envoy_stats_tags` - Specifies one or more static tags that will be added to
to `type.googleapis.com/envoy.api.v2.Listener`, or it may be the direct encoding all metrics produced by the proxy.
of the listener with no `@type` field.
Once parsed, it is passed to Envoy in place of the listener config that Consul - `envoy_stats_flush_interval` - Configures Envoy's
would typically configure. The only modifications Consul will make to the config [`stats_flush_interval`](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-stats-flush-interval).
provided are noted below.
#### Public Listener Configuration There are more possibilities available in the [Advanced
Configuration](#advanced-configuration) section that allow incremental or
complete control over the bootstrap configuration generated.
For the `proxy.config.envoy_public_listener_json`, every `FilterChain` added to ## Dynamic Configuration
the listener will have it's `TlsContext` overwritten with the Connect TLS
certificates. This means there is no way to override Connect TLS settings or the
requirement for all inbound clients to present valid Connect certificates.
Also, every `FilterChain` will have the `envoy.ext_authz` filter prepended to Consul automatically generates Envoy's dynamic configuration based on its
the filters array to ensure that all incoming connections must be authorized knowledge of the cluster. Users may specify default configuration options for
explicitly by the Consul agent based on their presented client certificate. each service such as which protocol they speak. Consul will use this information
to configure appropriate proxy settings for that service's proxies and also for
the upstream listeners of any downstream service.
To work properly with Consul Connect, the public listener should bind to the Users can define a service's protocol in its [`service-defaults` configuration
same address in the service definition so it is discoverable. It may also use entry](/docs/agent/config_entries.html#service-defaults-service-defaults). Agents with
the special cluster name `local_app` to forward requests to a single local [`enable_central_service_config`](/docs/agent/options.html#enable_central_service_config)
instance if the proxy was configured [as a set to true will automatically discover the protocol when configuring a proxy
sidecar](/docs/connect/proxies.html#sidecar-proxy-fields). for a service. The proxy will discover the main protocol of the service it
represents and use this to configure its main public listener. It will also
discover the protocols defined for any of its upstream services and
automatically configure its upstream listeners appropriately too as below.
#### Example This automated discovery results in Consul auto-populating the `proxy.config`
and `proxy.upstreams[*].config` fields of the [proxy service
definition](/docs/connect/proxies.html#proxy-service-definitions) that is
actually registered.
The following example shows a public listener being configured with an http ### Proxy Config Options
connection manager. As specified this behaves exactly like the default TCP proxy
filter however it provides metrics on HTTP request volume and response codes.
If additional config outside of the listener is needed (for example the These fields may also be overridden explicitly in the [proxy service
top-level `tracing` configuration to send traces to a collecting service), those definition](/docs/connect/proxies.html#proxy-service-definitions), or defined in
currently need to be added to a custom bootstrap. You may generate the default the [global `proxy-defaults` configuration
connect bootstrap with the [`consul connect envoy -bootstrap` entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) to act as
command](/docs/commands/connect/envoy.html) and then add the required additional defaults that are inherited by all services.
resources.
```text
service {
kind = "connect-proxy"
name = "web-http-aware-proxy"
port = 8080
proxy {
destination_service_name = "web"
destination_service_id = "web"
config {
envoy_public_listener_json = <<EOL
{
"@type": "type.googleapis.com/envoy.api.v2.Listener",
"name": "public_listener:0.0.0.0:18080",
"address": {
"socketAddress": {
"address": "0.0.0.0",
"portValue": 8080
}
},
"filterChains": [
{
"filters": [
{
"name": "envoy.http_connection_manager",
"config": {
"stat_prefix": "public_listener",
"route_config": {
"name": "local_route",
"virtual_hosts": [
{
"name": "backend",
"domains": ["*"],
"routes": [
{
"match": {
"prefix": "/"
},
"route": {
"cluster": "local_app"
}
}
]
}
]
},
"http_filters": [
{
"name": "envoy.router",
"config": {}
}
]
}
}
]
}
]
}
EOL
}
}
}
```
#### Upstream Listener Configuration - `protocol` - The protocol the service speaks. Connect's Envoy integration
currently supports the following `protocol` values:
- `tcp` - Unless otherwise specified this is the default, which causes Envoy
to proxy at L4. This provides all the security benefits of Connect's mTLS
and works for any TCP-based protocol. Load-balancing and metrics are
available at the connection level.
- `http` - This specifies that the service speaks HTTP/1.x. Envoy will setup an
`http_connection_manager` and will be able to load-balance requests
individually to available upstream services. Envoy will also emit L7 metrics
such as request rates broken down by HTTP response code family (2xx, 4xx, 5xx,
etc).
- `http2` - This specifies that the service speaks http2 (specifically h2c since
Envoy will still only connect to the local service instance via plain TCP not
TLS). This behaves much like `http` with L7 load-balancing and metrics but has
additional settings that correctly enable end-to-end http2.
- `grpc` - gRPC is a common RPC protocol based on http2. In addition to the
http2 support above, Envoy listeners will be configured with a
[gRPC bridge
filter](https://www.envoyproxy.io/docs/envoy/v1.9.1/configuration/http_filters/grpc_http1_bridge_filter#config-http-filters-grpc-bridge)
that translates HTTP/1.1 calls into gRPC, and instruments
metrics with `gRPC-status` trailer codes.
- `local_connect_timeout_ms` - The number of milliseconds allowed to make
connections to the local application instance before timing out. Defaults to 5000
(5 seconds).
For the upstream listeners `proxy.upstreams[].config.envoy_listener_json`, no ### Proxy Upstream Config Options
modification is performed. The `Clusters` served via the xDS API all have the
correct client certificates and verification contexts configured so outbound
traffic should be authenticated.
Each upstream may separately choose to define a custom listener config. If The following configuration items may be overridden directly in the
multiple upstreams define them care must be taken to ensure they all listen on `proxy.upstreams[].config` field of a [proxy service
separate ports. definition](/docs/connect/proxies.html#proxy-service-definitions) or
[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block.
Currently there is no way to disable a listener for an upstream, or modify how - `protocol` - Same as above in main config but affects the listener setup for
upstream service discovery clusters are delivered. Richer support for features the upstream.
like this is planned for the near future. - `connect_timeout_ms` - The number of milliseconds to allow when making upstream
connections before timing out. Defaults to 5000
(5 seconds).
## Advanced Configuration
To support more flexibility when configuring Envoy, several "lower-level" options exist that
exist that require knowledge of Envoy's configuration format.
Many allow configuring a subsection of either the bootstrap or
dynamic configuration using your own custom protobuf config.
We separate these into two sets, [Advanced Bootstrap
Options](#advanced-bootstrap-options) and [Escape Hatch
Overrides](#escape-hatch-overrides). Both require writing Envoy config in it's
protobuf JSON encoding. Advanced options are smaller chunks that might
commonly need to be set for tasks like configuring tracing. In contrast, escape hatches
give almost complete control over the proxy setup, but require operators to
manually code the entire configuration in protobuf JSON.
~> **Advanced Topic!** This section covers options that allow users to take almost
complete control of Envoy's configuration. We provide these options so users can
experiment or take advantage of features not yet fully supported in Consul Connect. We
plan to retain this ability in the future, but it should still be considered
experimental because it requires in-depth knowledge of Envoy's configuration format.
Users should consider Envoy version compatibility when using these features because they can configure Envoy in ways that
are outside of Consul's control. Incorrect configuration could prevent all
proxies in your mesh from functioning correctly, or bypass the security
guarantees Connect is designed to enforce.
### Configuration Formatting
All configurations are specified as strings containing the serialized proto3 JSON encoding
of the specified Envoy configuration type. They are full JSON types except where
noted.
The JSON supplied may describe a protobuf `types.Any` message with an `@type`
field set to the appropriate type (for example
`type.googleapis.com/envoy.api.v2.Listener`), or it may be the direct encoding
with no `@type` field.
### Advanced Bootstrap Options
Users may add the following configuration items to the [global `proxy-defaults`
configuration
entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) or
override them directly in the `proxy.config` field of a [proxy service
definition](/docs/connect/proxies.html#proxy-service-definitions) or
[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block.
- `envoy_extra_static_clusters_json` - Specifies one or more [Envoy
clusters](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/cds.proto#cluster)
that will be appended to the array of [static
clusters](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-staticresources-clusters)
in the bootstrap config. This allows adding custom clusters for tracing sinks
for example. For a single cluster just encode a single object, for multiple,
they should be comma separated with no trailing comma suitable for
interpolating directly into a JSON array inside the braces.
- `envoy_extra_static_listeners_json` - Similar to
`envoy_extra_static_clusters_json` but appends [static
listener](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-staticresources-listeners) definitions.
Can be used to setup limited access that bypasses Connect mTLS or
authorization for health checks or metrics.
- `envoy_extra_stats_sinks_json` - Similar to `envoy_extra_static_clusters_json`
but for [stats sinks](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-stats-sinks). These are appended to any sinks defined by use of the
higher-level [`envoy_statsd_url`](#envoy_statsd_url) or
[`envoy_dogstatsd_url`](#envoy_dogstatsd_url) config options.
- `envoy_stats_config_json` - The entire [stats
config](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-stats-config).
If provided this will override the higher-level
[`envoy_stats_tags`](#envoy_stats_tags). It allows full control over dynamic
tag replacements etc.
- `envoy_tracing_json` - The entire [tracing
config](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/config/bootstrap/v2/bootstrap.proto#envoy-api-field-config-bootstrap-v2-bootstrap-tracing).
Most tracing providers will also require adding static clusters to define the
endpoints to send tracing data to.
### Escape-Hatch Overrides
Users may add the following configuration items to the [global `proxy-defaults`
configuration
entry](/docs/agent/config_entries.html#proxy-defaults-proxy-defaults) or
override them directly in the `proxy.config` field of a [proxy service
definition](/docs/connect/proxies.html#proxy-service-definitions) or
[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block.
- `envoy_bootstrap_json_tpl` - Specifies a template in Go template syntax that
is used in place of [the default
template](https://github.com/hashicorp/consul/blob/b64bda880843afaaf44591c3200f921626716849/command/connect/envoy/bootstrap_tpl.go#L87)
when generating bootstrap via [`consul connect envoy`
command](/docs/commands/connect/envoy.html). The variables that are available
to be interpolated are [documented
here](https://github.com/hashicorp/consul/blob/b64bda880843afaaf44591c3200f921626716849/command/connect/envoy/bootstrap_tpl.go#L5).
This offers complete control of the proxy's bootstrap although major
deviations from the default template may break Consul's ability to correctly
manage the proxy or enforce it's security model.
- `envoy_public_listener_json` - Specifies a complete
[Listener](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/lds.proto)
to be delivered in place of the main public listener that the proxy used to
accept inbound connections. This will be used verbatim with the following
exceptions:
- Every `FilterChain` added to the listener will have its `TlsContext`
overridden by the Connect TLS certificates and validation context. This
means there is no way to override Connect's mutual TLS for the public
listener.
- Every `FilterChain` will have the `envoy.ext_authz` filter prepended to the
filters array to ensure that all inbound connections are authorized by
Connect.
- `envoy_local_cluster_json` - Specifies a complete [Envoy
cluster](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/cds.proto#cluster)
to be delivered in place of the local application cluster. This allows
customization of timeouts, rate limits, load balancing strategy etc.
The following configuration items may be overridden directly in the
`proxy.upstreams[].config` field of a [proxy service
definition](/docs/connect/proxies.html#proxy-service-definitions) or
[`sidecar_service`](/docs/connect/proxies/sidecar-service.html) block.
- `envoy_listener_json` - Specifies a complete
[Listener](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/lds.proto)
to be delivered in place of the upstream listener that the proxy exposes to
the application for outbound connections. This will be used verbatim with the
following exceptions:
- Every `FilterChain` added to the listener will have its `TlsContext`
overridden by the Connect TLS certificates and validation context. This
means there is no way to override Connect's mutual TLS for the public
listener.
- `envoy_cluster_json` - Specifies a complete [Envoy
cluster](https://www.envoyproxy.io/docs/envoy/v1.9.1/api-v2/api/v2/cds.proto#cluster)
to be delivered in place of the discovered upstream cluster. This allows
customization of timeouts, circuit breaking, rate limits, load balancing
strategy etc.