diff --git a/website/source/docs/guides/connect-services.html.md b/website/source/docs/guides/connect-services.html.md new file mode 100644 index 0000000000..3162dd79f3 --- /dev/null +++ b/website/source/docs/guides/connect-services.html.md @@ -0,0 +1,444 @@ +--- +name: Secure Service-to-Service Communication +content_length: 12 +id: connect-services +products_used: + - Consul +description: |- + This guide demonstrates Consul Connect using internal proxies as sidecars. +level: Implementation +--- + +Consul Connect secures service-to-service communication with authorization and +encryption. Applications can use sidecar proxies in a service mesh +configuration to automatically establish TLS connections for inbound and +outbound connections without being aware of Connect at all. In addition to +securing your services, Connect can also intercept [data about service-to-service +communications][consul-l7] and surface it to monitoring tools. + +In this guide, you will register two services and their [sidecar] proxies in +the Consul catalog. You will then start the services and sidecar proxies. +Finally, you will demonstrate that the service-to-service communication is going +through the proxies by stopping traffic with an "[intention]". + +[![Flow diagram showing end user traffic being sent to the Dashboard Service at +port 9002. The dashboard service makes requests for the counting service to the +local Connect Proxy at port 5000. This traffic then traverses the Connect mesh +over dynamic ports. The traffic exits the Connect mesh from the counting service's +local proxy. The proxy sends this traffic to the counting service itself at port +9003.][img-flow]][img-flow] + +While this guide uses elements that are not suitable for production +environments—Consul dev agents, internal proxies, and mock services—it will +teach you the common process for deploying your own services using Consul +Connect. At the end of this guide, we also present additional information +about adapting this process to more production-like environments. + +## Prerequisites + +To complete this guide, you will need a local [dev agent], which enables +Connect by default. + +This guide uses the following example service applications. Download and unzip +the executables to follow along. + +- [Counting Service] +- [Dashboard Service] + +### Verify your Consul agent health + +To ensure that Consul is running and accessible from the command line, use the +`consul members` command to verify your agent status. + +```shell +$ consul members +Node Address Status Type Build Protocol DC Segment +hostname.local 127.0.0.1:8301 alive server 1.6.1 2 dc1 +``` + +If you receive an error message, verify that you have a local Consul dev agent +running and try again. + +## Register the services and sidecar proxies + +Services have to be registered with Consul. Consul shares this information +around the cluster so that operators or other services can determine the +location of a service. Connect also uses service registrations to determine +where to send proxied traffic to. + +There are several ways to register services in Consul: + +- directly from a Consul-aware application + +- from an orchestrator, like [Nomad][services-nomad] or [Kubernetes][services-k8s] + +- [using configuration files][services-config] that are loaded at node startup + +- [using the API][services-api] to register them with a JSON or HCL + specification + +- [using the CLI][services-cli] to simplify this submission process + +For this guide, we will use the [`consul service register`][services-cli] CLI +command to load them into the catalog. + +### Create the counting service definition + +First, define the Counting service and its sidecar proxy in a file named +`counting.hcl`. The definition should include the name of the service, the port +the service listens on, and a [connect] block with the [sidecar_service] block. +This block is empty so Consul will use default parameters. The definition also +includes an optional service health check. + +```hcl +service { + name = "counting" + id = "counting-1" + port = 9003 + + connect { + sidecar_service {} + } + + check { + id = "counting-check" + http = "http://localhost:9003/health" + method = "GET" + interval = "1s" + timeout = "1s" + } +} +``` + +Services and sidecar proxies can be defined in either HCL or JSON. There is a +[JSON version][counting-1.json] version of the service definition in the +[demo-consul-101 project]. + +### Create the dashboard service definition + +Create the Dashboard service and proxy definition in the same way. First, +create a file named `dashboard.hcl`. + +```hcl +service { + name = "dashboard" + port = 9002 + + connect { + sidecar_service { + proxy { + upstreams = [ + { + destination_name = "counting" + local_bind_port = 5000 + } + ] + } + } + } + + check { + id = "dashboard-check" + http = "http://localhost:9002/health" + method = "GET" + interval = "1s" + timeout = "1s" + } +} +``` + +There is a [JSON version][dashboard.json] of the service definition in the +[demo-consul-101 project]. + +Notice that the dashboard definition also includes an upstream block. Upstreams +are ports on the local host that will be proxied to the destination service. +The upstream block's local_bind_port value is the port your service will +communicate with to reach the service you depend on. The destination name is +the Consul service name that the local_bind_port will proxy to. + +In our scenario, the dashboard service depends on the counting service. With +this configuration, when dashboard service connects to localhost:5000 it is +proxied across the service mesh to the counting service. + +### Register the services and proxies + +Finally, you can submit the service definitions to your Consul agent. If you +are using the JSON definitions, ensure that the filenames end in ".json" +instead of ".hcl". + +```shell +$ consul services register counting.hcl +Registered service: counting +``` + +```shell +$ consul services register dashboard.hcl +Registered service: dashboard +``` + +-> **Challenge:** After completing the guide, try doing it again using one of + the other service registration mechanisms mentioned earlier in the guide to + register the services. + +### Verify the services are registered + +Now that you have registered your services and sidecar proxies, run `consul +catalog services` to verify that they are present. + +```shell +$ consul catalog services +consul +counting +counting-sidecar-proxy +dashboard +dashboard-sidecar-proxy +``` + +### Create a Connect intention + +Intentions define access control for services via Connect and are used to +control which services may establish connections. The default intention +behavior is defined by the [default ACL policy]. + +In this guide, this step is not necessary since the default ACL policy of the +dev agent is "allow all", so Connect connections are automatically allowed as +well. However, we will create explicit intentions as a part of deploying +Connect-enabled services. + +-> **Best Practice:** Creating an explicit intention helps protect your + service against changes to the implied permissions. For example, a change in + `default_policy` or the introduction of a global deny-all intention would + impact services without explicit intentions defined. + +```shell +$ consul intention create dashboard counting +Created: dashboard => counting (allow) +``` + +## Start the services and sidecar proxies + +Now that you have created all the necessary configuration to describe your +service's connections, it's time to start your services and their sidecar +proxies. We are using the `&` operator to run the services as background tasks. +However, because they write to the console, it's best to run them in their own +shell session. + +Run these commands to start the applications: + +```shell +$ PORT=9002 COUNTING_SERVICE_URL="http://localhost:5000" ./dashboard-service & +$ PORT=9003 ./counting-service & +``` + +Next, start the sidecar proxies that will run as [sidecar] processes along with +the service applications. + +We are using the Consul Connect's built-in proxy for this guide. In a +production deployment, we recommend using Envoy instead. + +```shell +$ consul connect proxy -sidecar-for counting > counting-proxy.log & +$ consul connect proxy -sidecar-for dashboard > dashboard-proxy.log & +``` + +### Check the dashboard interface + +Open a browser and navigate to `http://localhost:9002`. + +You should see a screen similar to the following. There is a connection +indicator in the top right that will turn green and say "Connected" when the +dashboard service is in communication with the counting service. + +[![Image of Dashboard UI. There is white text on a magenta background, with the +page title "Dashboard" at the top left. There is a green indicator in the top +right with the word connected in white. There is a large number 19 to show +sample counting output. The node name that the counting service is running on, +host01, is in very small monospaced type underneath the large +numbers.][img-screenshot1]][img-screenshot1] + +If your application is not connected, check that the Counting service is +running and healthy by viewing it in the Consul UI at `http://localhost:8500`. + +## Test the sidecar proxy connections + +To test that traffic is flowing through the sidecar proxies, you will control +traffic with an intention. + +First, deny the Dashboard service access to the Counting service. + +```shell +$ consul intention create -deny -replace dashboard counting +Created: dashboard => counting (deny) +``` + +Refresh your browser, the connection indicator in the Dashboard ui will now say +"Disconnected" + +[![Image of Dashboard UI. There is white text on a magenta background, with the +page title "Dashboard" at the top left. There is a red indicator in the top +right with the words "Counting Service is Unreachable" in white. There is a +large number -1 to show sample counting output. The word "Unreachable" +surrounded by square brackets is in monospaced type underneath the large +numbers.][img-screenshot2]][img-screenshot2] + +You can restore communication between the services by replacing the `deny` +intention with an `allow`. + +```shell +$ consul intention create -allow -replace dashboard counting +``` + +Back in the browser, verify that the dashboard reconnects to the counting +service. + +## Clean up + +Once you are done with this guide, you can begin cleaning up by closing the +terminal in which your counting-service, dashboard-service, and proxies are +running. This should automatically stop these processes. + +Delete the intention from Consul. + +```shell +$ consul intention delete dashboard counting +Intention deleted. +``` + +Deregister the services. + +```shell +$ consul services deregister counting.hcl +Deregistered service: counting + +$ consul services deregister dashboard.hcl +Deregistered service: dashboard +``` + +## Extend these concepts + +When you want to apply this learning to a proof-of-concept or production +environment, there are some additional considerations. + +### Enable Connect and gRPC + +When Consul is started with the `-dev` flag, it will automatically enable +Consul Connect and provide a default port for gRPC communication. These have to +be configured explicitly for regular Consul agents. + +```hcl +# ... + +ports { + "grpc" = 8502 +} + +connect { + enabled = true +} + +``` + +For JSON configurations: + +```json +{ + // ... + "ports": { + "grpc": 8502 + }, + "connect": { + "enabled": true + } +} +``` + +### Download Envoy + +In this guide we used the built-in Connect proxy. For production deployments +and to enable L7 features, you should use Envoy. + +You can obtain container-based builds of Envoy directly from the [Envoy +Website], or you can obtain a packages of Envoy binary builds from a +third-party project, [getenvoy.io]. + +Consul will need to be able to find the "envoy" binary on the path. You can +extract the binary from the official Envoy Docker containers. + +To do this, create a container named "envoy-extract" based on the +"envoyproxy/envoy" container. + +```shell +$ docker create --name "envoy-extract" "envoyproxy/envoy" +docker create --name "envoy-extract" "envoyproxy/envoy" +Unable to find image 'envoyproxy/envoy:latest' locally +latest: Pulling from envoyproxy/envoy +16c48d79e9cc: Pull complete +3c654ad3ed7d: Pull complete +6276f4f9c29d: Pull complete +a4bd43ad48ce: Pull complete +ef9506777d3e: Pull complete +2e7ad8d4ceb7: Pull complete +d9e379d45dad: Pull complete +b283a3f5aebc: Pull complete +095fe71f6465: Pull complete +Digest: sha256:a7769160c9c1a55bb8d07a3b71ce5d64f72b1f665f10d81aa1581bc3cf850d09 +Status: Downloaded newer image for envoyproxy/envoy:latest +8d7bb45ea75f4344c6e050e5e1d3423937c4a1a51700ce34c3cf62a5ce3960dd +``` + +Use the `docker cp` command to copy the envoy file out of the container into +the current directory. + +```shell +$ docker cp "envoy-extract:/usr/local/bin/envoy" "envoy" +``` + +Now that you have the binary, you can remove the "envoy-extract" container. + +```shell +$ docker rm "envoy-extract" +envoy-extract +``` + +Once you have the binary extracted and in your path, Consul will automatically +use it when you run the `consul connect envoy` command. The following examples +demonstrate how to start the service sidecar proxies with Envoy. + +```shell +$ consul connect envoy -sidecar-for counting > counting-proxy.log & +$ consul connect envoy -sidecar-for dashboard > dashboard-proxy.log & +``` + +## Summary + +Now that you have completed this guide, you have familiarized yourself with a +basic Connect-enabled service deployment. You created and registered Consul +service definitions that describe how two services communicate with each other. +After starting the application and sidecar proxies, you used Consul Connect +intentions to control traffic flow between services. Finally, you learned about +additional requirements required to take the concepts to a proof-of-concept +environment. + +[connect]: https://www.consul.io/docs/connect/index.html +[consul-l7]: https://learn.hashicorp.com/consul/developer-mesh/l7-observability-k8s +[Counting Service]: https://github.com/hashicorp/demo-consul-101/releases/download/0.0.2/counting-service_linux_amd64.zip +[counting-1.json]:https://raw.githubusercontent.com/hashicorp/demo-consul-101/master/demo-config-localhost/counting-1.json +[Dashboard Service]: https://github.com/hashicorp/demo-consul-101/releases/download/0.0.2/dashboard-service_linux_amd64.zip +[dashboard.json]: https://raw.githubusercontent.com/hashicorp/demo-consul-101/master/demo-config-localhost/dashboard.json +[default ACL policy]: https://www.consul.io/docs/agent/options.html#acl_default_policy +[demo-consul-101 project]: https://github.com/hashicorp/demo-consul-101 +[dev agent]: https://learn.hashicorp.com/consul/getting-started/agent +[Docker guide]: https://learn.hashicorp.com/consul/day-0/containers-guide +[Envoy website]: https://www.envoyproxy.io/docs/envoy/latest/install/building#pre-built-binaries +[getenvoy.io]: https://www.getenvoy.io/ +[img-flow]: /static/img/consul/connect-getting-started/consul_connect_demo_service_flow.png +[img-screenshot1]: /static/img/consul/connect-getting-started/screenshot1.png +[img-screenshot2]: /static/img/consul/connect-getting-started/screenshot2.png +[intention]: https://www.consul.io/docs/connect/intentions.html +[services-api]: https://www.consul.io/api/agent/service.html#register-service +[services-cli]: https://www.consul.io/docs/commands/services.html +[services-config]: https://www.consul.io/docs/agent/services.html#service-definition +[services-nomad]: https://www.nomadproject.io/docs/job-specification/service.html +[sidecar]: https://docs.microsoft.com/en-us/azure/architecture/patterns/sidecar +[sidecar_service]: https://www.consul.io/docs/connect/registration/sidecar-service.html +[services-k8s]: https://www.consul.io/docs/platform/k8s/connect.html#installation-and-configuration