diff --git a/website/content/docs/lambda/index.mdx b/website/content/docs/lambda/index.mdx index 90b716ae41..51f46d6ee6 100644 --- a/website/content/docs/lambda/index.mdx +++ b/website/content/docs/lambda/index.mdx @@ -6,28 +6,42 @@ description: >- section documents the process of integrating AWS Lambda with Consul services. --- -# AWS Lambda +# AWS Lambda Overview -Lambda functions are programs or scripts that run in AWS Lambda. The functions process events and return responses. Refer to the [AWS Lambda website](https://aws.amazon.com/lambda/) for additional information. +You can configure Consul to allow services in your mesh to invoke Lambda functions, as well as allow Lambda functions to invoke services in your mesh. Lambda functions are programs or scripts that run in AWS Lambda. Refer to the AWS [Lambda website](https://aws.amazon.com/lambda/) for additional information. -## How AWS Lambda Functions on Consul Work +## Register Lambda functions into Consul -You can register AWS Lambda functions in Consul and invoke them from mesh services. -### Registering Lambda Functions +The first step is to register your Lambda functions into Consul. We recommend using the [Lambda registrator module](https://github.com/hashicorp/terraform-aws-consul-lambda/tree/main/modules/lambda-registrator) to automatically synchronize Lambda functions into Consul. You can also manually register Lambda functions into Consul if you are unable to use the Lambda registrator. -Registering AWS Lambda functions into Consul requires [registering a service](/docs/discovery/services) -and storing a [service defaults configuration entry](/docs/connect/config-entries/service-defaults) -into Consul. +Refer to [Register Lambda Functions Overview](TODO) for additional information about registering Lambda functions into Consul. -We recommend using [Lambda registrator](https://github.com/hashicorp/terraform-aws-consul-lambda-registrator) -to automatically synchronize Lambda functions into Consul. Lambda functions can -also be manually registered into Consul when using Lambda registrator is not possible. +## Invoke Lambda functions from Consul service mesh -See the [Registration page](/docs/lambda/registration) for more information -about registering Lambda functions into Consul. +After registering AWS Lambda functions, you can invoke Lambda functions from the Consul service mesh through terminating gateways (recommended) or directly from connect proxies. -### Invoking Lambda Functions from Consul Service Mesh +Refer to Invoke Lambda Functions from Services for details. -Lambda functions can be invoked by any mesh service directly from connect proxies or -through terminating gateways. The [Invocation page](/docs/lambda/invocation) -explains how to invoke Lambda functions from Consul service mesh services. +## Invoke mesh service from Lambda function + +You can also add the `consul-lambda-extension` plugin as a layer in your Lambda functions, which enables them to send requests to services in the mesh. The plugin starts a sidecar proxy that directs requests from Lambda functions to [mesh gateways](docs/connect/gateways#mesh-gateways). The gateways route traffic to the destination service to complete the request. + +![Invoke mesh service from Lambda function](/img/invoke-service-from-lambda-flow.svg) + +Refer to [Invoke Services from Lambda Functions](TODO) for additional information about registering Lambda functions into Consul. + +## Cross-datacenter communication + +You can use the following Consul features to send cross-datacenter requests between Lambda functions and mesh services. + +### Mesh gateway WAN federation + +Mesh gateways enable you to route traffic to services within and across Consul datacenters. WAN federation refers to designating a _primary datacenter_ that contains authoritative information about all datacenters, including service mesh configurations and access control list (ACL) resources. Refer to [Mesh Gateways for WAN Federation](/docs/connect/gateways/mesh-gateway/wan-federation-via-mesh-gateways) for additional information. + +Note that mesh gateways do not implement L7 traffic management by default. As a result, requests from Lambda functions ignore service routes and splitters. + +#### Admin partitions + +If admin partitions are enabled and the datacenters are federated across the WAN using mesh gateways, then you can only route requests from Lambda functions by applying an [`exported-services`](/docs/connect/config-entries/exported-services) configuration entry to export their service instances. This is required even if the upstream for the Lambda function is in the same admin partition. Otherwise, Consul does not populate the mesh gateways with the routing information. + +You can also use the [admin partitions](/docs/enterprise/partitions) feature included with Consul Enterprise to define separate administrative areas within a datacenter. If admin partitions are not enabled and the datacenters are federated across the WAN using mesh gateways, then you can route all services through the mesh gateways by default. You do not need to use the [`exported-services`](/docs/connect/config-entries/exported-services) configuration entry to export service instances. \ No newline at end of file diff --git a/website/content/docs/lambda/invocation.mdx b/website/content/docs/lambda/invocation.mdx index 4789c0adac..7bc91af239 100644 --- a/website/content/docs/lambda/invocation.mdx +++ b/website/content/docs/lambda/invocation.mdx @@ -5,7 +5,7 @@ description: >- This topic describes how to invoke AWS Lambda functions from the Consul service mesh. --- -# Invoke Lambda Functions +# Invoke Lambda Functions from Mesh Services This topic describes how to invoke AWS Lambda functions from the Consul service mesh. diff --git a/website/content/docs/lambda/invoke-from-lambda.mdx b/website/content/docs/lambda/invoke-from-lambda.mdx new file mode 100644 index 0000000000..6b00b347ca --- /dev/null +++ b/website/content/docs/lambda/invoke-from-lambda.mdx @@ -0,0 +1,301 @@ +--- +layout: docs +page_title: Invoke Services from Lambda Functions +description: >- + This topic describes how to invoke services in the mesh from Lambda functions registered with Consul. +--- + +# Invoke Services from Lambda Functions + +This topic describes how to invoke services in the mesh from Lambda functions registered with Consul. + +## Introduction + +The following steps describe the process: + +1. Deploy the services you want to allow Lambda to invoke. +1. (Optional) Enable L7 traffic management in the local datacenter. +1. Deploy the mesh gateway +1. Deploy the Lambda registrator +1. Invoke the Lambda function + +You must add the `consul-lambda-extension` extension as a Lambda layer to enable Lambda functions to send requests to mesh services. Refer to the [AWS Lambdas documentation](https://docs.aws.amazon.com/lambda/latest/dg/invocation-layers.html) for instructions on how to add layers to your Lambda functions. + +The layer runs an external Lambda extension that starts a sidecar proxy. The proxy listens on one port for each upstream service and upgrades the outgoing connections to mTLS. It then proxies the requests through to [mesh gateways](/docs/connect/gateways#mesh-gateways). + +## Prerequisites + +You must deploy the destination services and mesh gateway prior to deploying your Lambda service with the `consul-lambda-extension` layer. It’s not required, but you can also enable L7 traffic management in the local datacenter prior to implementing the `consul-lambda-extension` layer. + +### Deploy the destination service + +There are several methods for deploying services to Consul service mesh. The following example configuration deploys a service named `static-server` with Consul on Kubernetes. + +```yaml +kind: Service +apiVersion: v1 +metadata: + # Specifies the service name in Consul. + name: static-server +spec: + selector: + app: static-server + ports: + - protocol: TCP + port: 80 + targetPort: 8080 +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: static-server +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: static-server +spec: + replicas: 1 + selector: + matchLabels: + app: static-server + template: + metadata: + name: static-server + labels: + app: static-server + annotations: + 'consul.hashicorp.com/connect-inject': 'true' + spec: + containers: + - name: static-server + image: hashicorp/http-echo:latest + args: + - -text="hello world" + - -listen=:8080 + ports: + - containerPort: 8080 + name: http + serviceAccountName: static-server +``` + +### Enable L7 traffic management (optional) + +Mesh gateways do not implement L7 traffic management features, but you can enable L7 in the local data center so that your service can use service resolvers, splitters, and routers. + +1. Define an `exported-services` configuration entry. Refer to [Exported Services](/docs/connect/config-entries/exported-services) for additional information. The following example exports `static-server` service instances to a peered cluster specified in the `PeerName` field. + + + + ```hcl + Kind = "exported-services" + Name = "default" + Services = [ + { + Name = "static-server" + Consumers = [ + { + PeerName = "" + } + ] + } + ] + ``` + + + +1. Apply the configuration using the Consul CLI or by using a custom resource definition (CRD) if Consul is running on Kubernetes. The following example shows the command line usage: + + ```shell-session + $ consul config write static-server-configuration-entry.hcl + ``` +### Deploy the mesh gateway + +The mesh gateway must be running and registered to the Lambda function’s Consul datacenter. Refer to the following documentation and tutorials for instructions: + +- (Mesh Gateways between Datacenters)(/docs/connect/gateways/mesh-gateway/service-to-service-traffic-datacenters) +- [Mesh Gateways between Admin Partitions](/docs/connect/gateways/mesh-gateway/service-to-service-traffic-partitions) +- [Mesh Gateways between Peered Clusters](/docs/connect/gateways/mesh-gateway/service-to-service-traffic-peers) +- [Connect Services Across Datacenters with Mesh Gateways](https://developer.hashicorp.com/consul/tutorials/developer-mesh/service-mesh-gateways) + +## Deploy the Lambda extension layer + +The `consul-lambda-extension` extension runs during the `init` phase of the Lambda function execution. The extension retrieves the data that the Lambda registrator has been configured to store from AWS Parameter Store and creates a lightweight TCP proxy. The proxy creates a local listener for each upstream defined in the `CONSUL_SERVICE_UPSTREAMS` environment variable. + +When the Lambda function is invoked, the extension retrieves the data from the AWS Parameter Store so that the function can process requests. When the Lambda function receives a shutdown event, the extension also stops. + +1. Download the `consul-lambda-extension` extension from releases.hashicorp.com: + + ```shell-session + curl -o consul-lambda-extension__linux_amd64.zip https://releases.hashicorp.com/consul-lambda//consul-lambda-extension__linux_amd64.zip + ``` +1. Create the AWS Lambda layer. You can create the layer manually using the AWS CLI or AWS Console, but we recommend using Terraform: + + + + ``` + resource "aws_lambda_layer_version" "consul_lambda_extension" { + layer_name = "consul-lambda-extension" + filename = "consul-lambda-extension__linux_amd64.zip" + source_code_hash = filebase64sha256("consul-lambda-extension__linux_amd64.zip") + description = "Consul service mesh extension for AWS Lambda" + } + ``` + + + +## Deploy the Lambda registrator + +Configure and deploy the Lambda registrator. Refer to the [registrator configuration documentation](/docs/lambda/registration/automate#configuration) and the [registrator deployment documentation](/docs/lambda/registration/automate#deploy-the-lambda-registrator) for instructions. + +## Write the Lambda function code + +Refer to the [AWS Lambda documentation](https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html) for instructions on how to write a Lambda function. In the following example, the function calls an upstream service on port `2345`: + + +```go +package main + +import ( + "context" + "io" + "fmt" + "net/http" + "github.com/aws/aws-lambda-go/lambda" +) + +type Response struct { + StatusCode int `json:"statusCode"` + Body string `json:"body"` +} + +func HandleRequest(ctx context.Context, _ interface{}) (Response, error) { + resp, err := http.Get("http://localhost:2345") + fmt.Println("Got response", resp) + if err != nil { + return Response{StatusCode: 500, Body: "Something bad happened"}, err + } + + if resp.StatusCode != 200 { + return Response{StatusCode: resp.StatusCode, Body: resp.Status}, err + } + + defer resp.Body.Close() + + b, err := io.ReadAll(resp.Body) + if err != nil { + return Response{StatusCode: 500, Body: "Error decoding body"}, err + } + + return Response{StatusCode: 200, Body: string(b)}, nil +} + +func main() { + lambda.Start(HandleRequest) +} +``` + +## Deploy the Lambda function + +1. Create and apply an IAM policy that allows the Lambda function’s role to fetch the Lambda extension’s data from the Parameter Store. The following example, creates an IAM role for the Lambda function, creates an IAM policy with the necessary permissions and attaches the policy to the role: + + + + ```hcl + resource "aws_iam_role" "lambda" { + name = "lambda-role" + + assume_role_policy = < + +1. Configure and deploy the Lambda function. Refer to the [Lambda function configuration](#lambda-function-configuration) reference for information about all available options. There are several methods for deploying Lambda functions. The following example uses Terraform to deploy a function that can invoke the `static-server` upstream service using mTLS data stored under the `/lambda_extension_data` prefix: + + + + ```hcl + resource "aws_lambda_function" "example" { + … + function_name = "lambda" + role = aws_iam_role.lambda.arn + tags = { + "serverless.consul.hashicorp.com/v1alpha1/lambda/enabled" = "true" + } + variables = { + environment = { + CONSUL_MESH_GATEWAY_URI = var.mesh_gateway_http_addr + CONSUL_SERVICE_UPSTREAMS = "static-server:2345:dc1" + CONSUL_EXTENSION_DATA_PREFIX = “/lambda_extension_data” + } + } + layers = [aws_lambda_layer_version.consul_lambda_extension.arn] + ``` + + + +1. Issue the `terraform apply` command and Consul automatically configures a service for the Lambda function. + + +### Lambda function configuration + +Define the following environment variables to configure each Lambda function. The configurations apply to each Lambda function in your environment: + +| Variable | Description | Default | +| --- | --- | --- | +| `CONSUL_MESH_GATEWAY_URI` | Specifies the URI where the mesh gateways that the plugin makes requests are running. The mesh gateway should be registered in the same Consul datacenter and partition that the service is running in. For optimal performance, this mesh gateway should run in the same AWS region. | none | +| `CONSUL_EXTENSION_DATA_PREFIX` | Specifies the prefix that the plugin pulls configuration data from. The data must be located in the following directory:
`“${CONSUL_EXTENSION_DATA_PREFIX}/${CONSUL_SERVICE_PARTITION}/${CONSUL_SERVICE_NAMESPACE}/”` | none | +| `CONSUL_SERVICE_NAMESPACE` | Specifies the Consul namespace the service is registered into. | `default` | +| `CONSUL_SERVICE_PARTITION` | Specifies the Consul partition the service is registered into. | `default` | +| `CONSUL_REFRESH_FREQUENCY` | Specifies the amount of time the extension waits before re-pulling data from the Parameter Store. Use [Go `time.Duration`](https://pkg.go.dev/time@go1.19.1#ParseDuration) string values, for example, `”30s”`.
The time is added to the duration configured in the Lambda registrator `sync_frequency_in_minutes` configuration. Refer to [Lambda registrator configuration options](/docs/lambda/registration/automate#lambda-registrator-configuration-options). The combined configurations determine how stale the data may become. Lambda functions can run for up to 14 hours. We recommend configuring an acceptable value to preview stale certificates. | `“5m”` | +| `CONSUL_SERVICE_UPSTREAMS` | Specifies the upstream services that the Lambda function can call. Specify the value as an unlabelled annotation according to the [`consul.hashicorp.com/connect-service-upstreams` annotation format](/docs/k8s/annotations-and-labels#consul-hashicorp-com-connect-service-upstreams) in Consul on Kubernetes. For example, `"[service-name]:[port]:[optional-datacenter]"` | none | + +## Invoke the Lambda function + +You can create an _intention_ in Consul prior to invoking the Lambda function. Intentions define access control for services in the mesh. Refer to [Service Mesh Intentions](/docs/connect/intentions) for additional information. + +There are several ways to invoke Lambda functions. In the following example, the `aws lambda invoke` CLI command invokes the function.: + +```shell-session +$ aws lambda invoke --function-name lambda-registrator-2345 /dev/stdout | cat +``` diff --git a/website/content/docs/lambda/registration.mdx b/website/content/docs/lambda/registration.mdx deleted file mode 100644 index 9fe9ba0da5..0000000000 --- a/website/content/docs/lambda/registration.mdx +++ /dev/null @@ -1,290 +0,0 @@ ---- -layout: docs -page_title: Register Lambda Functions -description: >- - This topic describes how to register AWS Lambda functions with Consul service mesh. ---- - -# Register Lambda Functions - -You can either manually register AWS Lambda functions with Consul or use the [Lambda registrator](https://github.com/hashicorp/terraform-aws-consul-lambda-registrator) -to automatically synchronize Lambda state into Consul. - -To manually register AWS Lambda functions into Consul, you must register a service into Consul and then write a [service defaults configuration entry](/docs/connect/config-entries/service-defaults) for the Lambda. - -The registrator automatically registers, reconfigures, and deregisters Lambdas based on the -Lambda function's tags (refer to the [AWS tag configuration documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-tags.html) for details about tags). - -We recommend using the Lambda registrator when possible so that you can keep the configuration entry up to date. - -## Requirements - -- Consul 1.12.1 and later - -## Prerequisites - -Complete the following prerequisites prior to registering your Lambda functions. You only need to perform these steps once. - -### Enable the Serverless Plugin - -Add the following configuration to all Consul clients: - -`connect { enable_serverless_plugin = true, connect = true }` - -Refer to the [`enable_serverless_plugin`](/docs/agent/config/config-files#connect_enable_serverless_plugin) configuration documentation for additional information. - -### Configure IAM Permissions for Envoy - -The Envoy proxy that invokes Lambda must have the `lambda:InvokeFunction` AWS IAM -permissions. In the following example, the IAM policy -enables an IAM user or role to invoke the `example` Lambda function: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "Invoke", - "Effect": "Allow", - "Action": [ - "lambda:InvokeFunction" - ], - "Resource": "arn:aws:lambda:us-east-1:123456789012:function:example" - } - ] -} -``` - -Define AWS IAM credentials in environment variables, EC2 metadata or -ECS metadata. On [AWS EKS](https://aws.amazon.com/eks/), associate an IAM role with the proxy's `ServiceAccount`. Refer to the [AWS IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) documentation for instructions. - -### Optional: Set up a Terminating Gateway - -If you intend to invoke Lambda services through a terminating gateway, the gateway must be registered and running in the Consul datacenter. Refer to the following documentation and tutorials for instructions on how to set up a terminating gateway: - -- [Terminating gateways documentation](/docs/connect/gateways#terminating-gateways) -- [Terminating gateways on Kubernetes documentation](/docs/k8s/connect/terminating-gateways) -- [Connect External Services to Consul With Terminating Gateways tutorial](https://learn.hashicorp.com/tutorials/consul/teminating-gateways-connect-external-services) - -To register a Lambda service with a terminating gateway, add the service to the -`Services` field of the terminating gateway's `terminating-gateway` -configuration entry. - -### Optional: Run a Mesh Gateway - -You can set up a mesh gateway so that you can invoke Lambda services across datacenters and admin partitions. The mesh gateway must be running and registered in the relevant Consul datacenters and partitions. Refer to the following documentation and tutorials for instructions on how to set up mesh gateways: - -- [Mesh gateway documentation](/docs/connect/gateways#mesh-gateways) -- [Connect Services Across Datacenters with Mesh Gateways tutorial](https://learn.hashicorp.com/tutorials/consul/service-mesh-gateways) -- [Secure Service Mesh Communication Across Kubernetes Clusters tutorial](https://learn.hashicorp.com/tutorials/consul/kubernetes-mesh-gateways?utm_source=docs?in=consul/kubernetes) - -When using admin partitions, you must add Lambda services to the `Services` -field of [the `exported-services` configuration -entry](/docs/connect/config-entries/exported-services). - -## Automatic Lambda Function Registration - -You can deploy the Lambda registrator to your environment to automatically register and deregister Lambda functions with Consul based on the function's tags. Refer to the [AWS Lambda tags documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-tags.html) to learn about tags. - -The registrator runs as a Lambda function that is invoked by AWS EventBridge. Refer to the [AWS EventBridge documentation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) for additional information. - -EventBridge invokes the registrator using either [AWS CloudTrail](https://docs.aws.amazon.com/lambda/latest/dg/logging-using-cloudtrail.html) to synchronize with Consul in real-time or in [scheduled intervals](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule-schedule.html). - -CloudTrail events typically synchronize updates, registration, and deregistration within one minute, but events may occasionally be delayed. - -Scheduled events fully synchronize functions between Lambda and Consul to prevent entropy. By default, EventBridge triggers a full sync every five minutes. - -The following diagram shows the flow of events from EventBridge into Consul: - - - -![Lambda Registrator Architecture](/img/lambda_registrator_architecture.svg) - - - -1. EventBridge invokes the Lambda registrator based on CloudTrail Lambda events or a schedule. -1. Lambda registrator determines how to reconcile Lambda's control plane state - with Consul state and ensures they are in sync by registering, updating, and - deregistering Lambda services. - -### Deploy Lambda Registrator - -1. Create a Terraform configuration and specify the `lambda-registrator` module. In the following example, the Lambda registrator is deployed to `https://consul.example.com:8501`. Refer to [the Lambda registrator module documentation](https://registry.terraform.io/modules/hashicorp/consul-lambda-registrator/aws/0.1.0-beta1/submodules/lambda-registrator) for additional usage information: - ```hcl - module "lambda-registrator" { - source = "hashicorp/consul-lambda-registrator/aws//modules/lambda-registrator" - name = "consul-lambda-registrator" - consul_http_addr = "https://consul.example.com:8501" - ca_cert_path = aws_ssm_parameter.ca-cert.name - http_token_path = aws_ssm_parameter.acl-token.name - } - ``` - -1. Deploy Lambda registrator with `terraform apply`. - -#### Optional: Store the CA Certificate in Parameter Store - -When Lambda registrator makes a request to Consul's [HTTP API](/api-docs) over HTTPS and the Consul API is signed by a custom CA, Lambda registrator uses the CA certificate stored in AWS Parameter Store (refer to the [Parameter Store documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) for additional information) to verify the authenticity of the Consul API. You can apply the following Terraform configuration to store Consul's server CA in Parameter Store: - -```hcl -resource "aws_ssm_parameter" "ca-cert" { - name = "/lambda-registrator/ca-cert" - type = "SecureString" - value = -} -``` - -#### Optional: Store the ACL Token in Parameter Store - -If [Consul access control lists (ACLs)](/docs/security/acl) are enabled, Lambda registrator must present an ACL token stored in Parameter Store to access resources. You can use the Consul CLI, API, or the Terraform provider to facilitate the ACL workflow. The following procedure describes how to create and store a token from the command line: - -1. Create an ACL policy that includes the following rule: - - - - ```hcl - service_prefix "" { - policy = "write" - } - ``` - - - -1. Issue `consul acl policy create` command to create the policy. The following example creates a policy called `lambda-registrator-policy` containing permissions specified in `rules.hcl`: - ```shell-session - $ consul acl policy create -name "lambda-registrator-policy" -rules @rules.hcl - ``` - -1. Issue the `consul acl token create` command to create the token. The following example creates a token linked to the `lambda-registrator-policy` policy: - ```shell-session - $ consul acl token create -policy-name "lambda-registrator-policy" - ``` - -1. Store the token in Parameter Store by applying the following Terraform: - ```hcl - resource "aws_ssm_parameter" "acl-token" { - name = "/lambda-registrator/acl-token" - type = "SecureString" - value = - } - ``` - -#### Lambda Registrator Configuration Options - -| Name | Description | -| - | - | -| `name` | Specifies the name name of the Lambda function associated with the Lambda registrator. The name is also used to construct the Identity and Access Management (IAM) role and policy names used by the Lambda function. | -| `schedule_frequency_in_minutes` | Specifies the interval in minutes that EventBridge uses to trigger a full synchronization. Default is `5`. | -| `timeout` | The maximum number of seconds Lambda registrator can run per invocation before timing out. | -| `consul_http_addr` | Specifies the address of the Consul API client. | -| `consul_ca_cert_path` | Specifies the path to the CA certificate stored in the AWS Parameter Store. When Lambda registrator makes an HTTPS request to Consul's API and the Consul API is signed by a custom CA, Lambda registrator uses this CA certificate to verify the authenticity of the Consul API. At startup, Lambda registrator pulls the CA certificate at this path from Parameter Store, writes the certificate to the filesystem and stores the path of that file in `CONSUL_CACERT`. Also see [Optional: Store the CA Certificate in Parameter Store](#optional-store-the-ca-certificate-in-parameter-store)| -| `consul_http_token_path` | Specifies the path to the ACL token stored in AWS Parameter Store that Lambda registrator presents to access resources. This parameter only required when ACLs are enabled for the Consul server. It is used to fetch an ACL token from Parameter Store and is stored in the `CONSUL_HTTP_TOKEN` environment variable. Also see [Optional: Store the ACL Token in Parameter Store](#optional-store-the-acl-token-in-parameter-store)| -| `node_name` | The Consul node name that Lambdas will be registered to. This defaults to `lambdas`. | -| `enterprise` | Determines if the Consul server at `consul_http_addr` is running open source or enterprise. | -| `partitions` | The partitions that Lambda registrator manages. | - -### Register Lambda Functions - -Lambda registrator registers Lambda functions into Consul, regardless of how the functions are -deployed. The following procedure describes how to register Lambda functions with the Lambda registrator using Terraform, but you can also deploy a Lambda function with CloudFormation, the AWS user -interface, or Cloud Development Kit (CDK): - -1. Add the `aws_lambda_function` resource to your Terraform configuration and specify the name of the Lambda function. -1. Add a `tags` block to the resource and specify the tags you want to use to register the function (refer to [Supported Tags](#supported-tags)). - -In the following example, the `example` Lambda function is registered using the `enabled`, `payload-passthrough`, and `invocation-mode` tags: - -```hcl -resource "aws_lambda_function" "example" { - … - function_name = "lambda" - tags = { - "serverless.consul.hashicorp.com/v1alpha1/lambda/enabled" = "true" - "serverless.consul.hashicorp.com/alpha/lambda/payload-passthrough" = "true" - "serverless.consul.hashicorp.com/alpha/lambda/invocation-mode" = "ASYNCHRONOUS" - } -} -``` - -#### Supported Tags - -The following tags are supported. In all cases, the `` should be -`serverless.consul.hashicorp.com/v1alpha1/lambda`. For example, -`serverless.consul.hashicorp.com/v1alpha1/lambda/enabled`. - -| Tag | Description | -| - | - | -| `/enabled` | Determines if Lambda registrator will sync the Lambda into Consul. | -| `/payload-passthrough` | Determines if the body Envoy receives is converted to JSON or directly passed to Lambda. This attribute is optional and defaults to `false`. | -| `/invocation-mode` | Specifies the [Lambda invocation mode](https://docs.aws.amazon.com/lambda/latest/operatorguide/invocation-modes.html) Consul uses to invoke the Lambda. The default is `SYNCHRONOUS`, but `ASYNCHRONOUS` invocations are also supported. | -| `/namespace` | Specifies the Consul namespace the service will be registered in. Default is `default` if `enterprise` is enabled. | -| `/partition` | Specifies the Consul partition the service will be registered in. Defaults is `default` if `enterprise` is enabled. | -| `/aliases` | Specifies a `+`-separated string of Lambda aliases that will be registered into Consul. For example, if set to `dev+staging+prod`, the `dev`, `staging`, and `prod` aliases of the Lambda function will be registered into Consul. | - -## Manual Configuration - -You can manually register Lambda functions if you are unable to automate the process using the Lambda registrator. - -1. Create a configuration for registering the service. You can copy the following example and replace `` with your Consul service name for the Lambda function: - - - - ```json - { - "Node": "lambdas", - "SkipNodeUpdate": true, - "NodeMeta": { - "external-node": "true", - "external-probe": "true" - }, - "Service": { - "Service": "" - } - } - ``` - - - -1. Save the configuration to `lambda.json`. - -1. Send the configuration to the `catalog/register` API endpoint to register the service, for example: - ```shell-session - $ curl --request PUT --data @lambda.json localhost:8500/v1/catalog/register - ``` - -1. Create the `service-defaults` configuration entry and include the AWS tags used to invoke the Lambda function in the `Meta` field (see [Supported `Meta` Fields](#supported-meta-fields). The following example creates a `service-defaults` configuration entry named `lambda`: - - - - ```hcl - Kind = "service-defaults" - Name = "lambda" - Protocol = "http" - Meta = { - "serverless.consul.hashicorp.com/v1alpha1/lambda/enabled" = "true" - "serverless.consul.hashicorp.com/v1alpha1/lambda/arn" = "" - "serverless.consul.hashicorp.com/v1alpha1/lambda/payload-passthrough" = "true" - "serverless.consul.hashicorp.com/v1alpha1/lambda/region" = "us-east-2" - } - ``` - - - -1. Issue the `consul config write` command to store the configuration entry. For example: - ```shell-session - $ consul config write lambda-service-defaults.hcl - ``` - -### Supported `Meta` Fields - -The following tags are supported. In all cases, the `` should be -`serverless.consul.hashicorp.com/v1alpha1/lambda`. For example, -`serverless.consul.hashicorp.com/v1alpha1/lambda/enabled`. - -| Tag | Description | -| - | - | -| `/enabled` | Determines if Consul configures the service as an AWS Lambda. | -| `/payload-passthrough` | Determines if the body Envoy receives is converted to JSON or directly passed to Lambda. | -| `/arn` | Specifies the [AWS ARN](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) for the service's Lambda. | -| `/invocation-mode` | Determines if Consul configures the Lambda to be invoked using the `synchronous` or `asynchronous` [invocation mode](https://docs.aws.amazon.com/lambda/latest/operatorguide/invocation-modes.html). | -| `/region` | Specifies the AWS region the Lambda is running in. | diff --git a/website/content/docs/lambda/registration/automate.mdx b/website/content/docs/lambda/registration/automate.mdx new file mode 100644 index 0000000000..0c8d82d183 --- /dev/null +++ b/website/content/docs/lambda/registration/automate.mdx @@ -0,0 +1,190 @@ +--- +layout: docs +page_title: Automate Lambda Function Registeration +description: >- + Register AWS Lambda functions with Consul service mesh using the Consul Lambda registrator. The Consul Lambda registrator automates Lambda function registration. +--- + +# Automate Lambda Function Registeration + +This topic describes how to automate Lambda function registration using the Consul Lambda registrator module for Terraform. + +## Introduction + +You can deploy the Lambda registrator to your environment to automatically register and deregister Lambda functions with Consul based on the function's tags. Refer to the [AWS Lambda tags documentation](https://docs.aws.amazon.com/lambda/latest/dg/configuration-tags.html) to learn about tags. + +The registrator also stores and periodically updates information required to make mTLS requests to upstream services in the AWS parameter store. This enables Lambda functions to invoke mesh services. Refer to [Invoke Services from Lambda Functions](TODO) for additional information. + +The registrator runs as a Lambda function that is invoked by AWS EventBridge. Refer to the [AWS EventBridge documentation](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html) for additional information. + +EventBridge invokes the registrator using either [AWS CloudTrail](https://docs.aws.amazon.com/lambda/latest/dg/logging-using-cloudtrail.html) to synchronize with Consul in real-time or in scheduled intervals. + +CloudTrail events typically synchronize updates, registration, and deregistration within one minute, but events may occasionally be delayed. + +Scheduled events fully synchronize functions between Lambda and Consul to prevent entropy. By default, EventBridge triggers a full sync every five minutes. + +The following diagram shows the flow of events from EventBridge into Consul: + + + + +![Lambda Registrator Architecture](/img/lambda_registrator_architecture.svg) + + + +1. EventBridge invokes the Lambda registrator based on CloudTrail Lambda events or a schedule. +1. Lambda registrator determines how to reconcile Lambda's control plane state + with Consul state and ensures they are in sync by registering, updating, and + deregistering Lambda services. + +## Requirements + +Verify that your environment meets the requirements specified in [Lambda Function Registration Requirements](/docs/lambda/registration/index). + +## Configuration + +The Lambda registrator module stores data in the AWS parameter store. You can configure the type of data stored and how to store it. + +### Optional: Store the CA certificate in Parameter Store + +When Lambda registrator makes a request to Consul's [HTTP API](/api-docs) over HTTPS and the Consul API is signed by a custom CA, Lambda registrator uses the CA certificate stored in AWS Parameter Store (refer to the [Parameter Store documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html) for additional information) to verify the authenticity of the Consul API. You can apply the following Terraform configuration to store Consul's server CA in Parameter Store: + +```hcl +resource "aws_ssm_parameter" "ca-cert" { + name = "/lambda-registrator/ca-cert" + type = "SecureString" + value = +} +``` + +### Optional: Store the ACL token in Parameter Store + +If [Consul access control lists (ACLs)](/docs/security/acl) are enabled, Lambda registrator must present an ACL token stored in Parameter Store to access resources. You can use the Consul CLI, API, or the Terraform provider to facilitate the ACL workflow. The following procedure describes how to create and store a token from the command line: + +1. Create an ACL policy that includes the following rule: + + + + ```hcl + service_prefix "" { + policy = "write" + } + ``` + + + +1. Issue `consul acl policy create` command to create the policy. The following example creates a policy called `lambda-registrator-policy` containing permissions specified in `rules.hcl`: + ```shell-session + $ consul acl policy create -name "lambda-registrator-policy" -rules @rules.hcl + ``` + +1. Issue the `consul acl token create` command to create the token. The following example creates a token linked to the `lambda-registrator-policy` policy: + ```shell-session + $ consul acl token create -policy-name "lambda-registrator-policy" + ``` + +1. Store the token in Parameter Store by applying the following Terraform: + ```hcl + resource "aws_ssm_parameter" "acl-token" { + name = "/lambda-registrator/acl-token" + type = "SecureString" + value = + } + ``` + +### Optional: Store extension data in Parameter Store + +If you want to enable Lambda functions to invoke services in the mesh, then you must specify a non-empty string in the `consul_extension_data_prefix` configuration. The string represents a path in the AWS Parameter Store so that the registrator can store data necessary for making mTLS requests to upstream services and update the data periodically. If the path does not exist it will be created. + +Lambda registrator encrypts and stores all data for Lambda functions in the AWS Parameter Store according to the [Lambda registrator configuration options](#lambda-registrator-configuration-options). The data is stored in the following directory as a `SecureString` type: + +`${var.consul_extension_data_prefix}/${}/${}/${}` + +The registrator also requires the following IAM permissions to access the parameter store: + + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": ["ssm:PutParameter","ssm:DeleteParameter"], + "Resource": "arn:aws:ssm:us-east-2:123456789012:parameter/${var.consul_extension_data_prefix}/*" + }, + ] +} +``` + +### Lambda Registrator configuration options + +| Name | Description | +| - | - | +| `name` | Specifies the name name of the Lambda function associated with the Lambda registrator. The name is also used to construct the Identity and Access Management (IAM) role and policy names used by the Lambda function. | +| `schedule_frequency_in_minutes` | Specifies the interval in minutes that EventBridge uses to trigger a full synchronization. Default is `10`. | +| `timeout` | The maximum number of seconds Lambda registrator can run per invocation before timing out. | +| `consul_http_addr` | Specifies the address of the Consul API client. | +| `consul_datacenter` | Specifies the Consul datacenter to synchronize with AWS Lambda state data. By default, the Lambda registrator manages Lambda services for all Consul datacenters. When configured for a specific datacenter, Lambda registrator only manages Lambda services with a matching datacenter tag. Refer to [Supported tags](#supported-tags) for additional information. | +| `consul_extension_data_prefix` | Specifies the path prefix in the AWS Parameter Store under which the registrator manages mTLS data. If Lambda functions call mesh services, the value must be set to a non-empty string starting with `/`. | +| `consul_ca_cert_path` | Specifies the path to the CA certificate stored in the AWS Parameter Store. When Lambda registrator makes an HTTPS request to Consul's API and the Consul API is signed by a custom CA, Lambda registrator uses this CA certificate to verify the authenticity of the Consul API. At startup, Lambda registrator pulls the CA certificate at this path from Parameter Store, writes the certificate to the filesystem and stores the path of that file in `CONSUL_CACERT`. Also see [Optional: Store the CA Certificate in Parameter Store](#optional-store-the-ca-certificate-in-parameter-store)| +| `consul_http_token_path` | Specifies the path to the ACL token stored in AWS Parameter Store that Lambda registrator presents to access resources. This parameter only required when ACLs are enabled for the Consul server. It is used to fetch an ACL token from Parameter Store and is stored in the `CONSUL_HTTP_TOKEN` environment variable. Also see [Optional: Store the ACL Token in Parameter Store](#optional-store-the-acl-token-in-parameter-store)| +| `node_name` | The Consul node name that Lambdas will be registered to. This defaults to `lambdas`. | +| `enterprise` | Determines if the Consul server at `consul_http_addr` is running open source or enterprise. | +| `partitions` | The partitions that Lambda registrator manages. | + +## Deploy the Lambda registrator + +1. Create a Terraform configuration and specify the `lambda-registrator` module. In the following example, the Lambda registrator is deployed to `https://consul.example.com:8501`. Refer to [the Lambda registrator module documentation](https://registry.terraform.io/modules/hashicorp/consul-lambda-registrator/aws/0.1.0-beta1/submodules/lambda-registrator) for additional usage information: + ```hcl + module "lambda-registrator" { + source = "hashicorp/consul-lambda/consul-lambda-registrator" + version = "x.y.z" + name = "consul-lambda-registrator" + consul_http_addr = “https://aecfe39d629774e348a9844439f5e3c1-1471365273.us-east-1.elb.amazonaws.com:8501” + ca_cert_path = aws_ssm_parameter.ca-cert.name + http_token_path = aws_ssm_parameter.acl-token.name + consul_extension_data_prefix = “/lambda_extension_data” + } + ``` + +1. Deploy Lambda registrator with `terraform apply`. + + +## Register Lambda functions + +Lambda registrator registers Lambda functions into Consul, regardless of how the functions are +deployed. The following procedure describes how to register Lambda functions with the Lambda registrator using Terraform, but you can also deploy a Lambda function with CloudFormation, the AWS user +interface, or Cloud Development Kit (CDK): + +1. Add the `aws_lambda_function` resource to your Terraform configuration and specify the name of the Lambda function. +1. Add a `tags` block to the resource and specify the tags you want to use to register the function (refer to [Supported tags](#supported-tags)). + +In the following example, the `example` Lambda function is registered using the `enabled`, `payload-passthrough`, and `invocation-mode` tags: + +```hcl +resource "aws_lambda_function" "example" { + … + function_name = "lambda" + tags = { + "serverless.consul.hashicorp.com/v1alpha1/lambda/enabled" = "true" + "serverless.consul.hashicorp.com/alpha/lambda/payload-passthrough" = "true" + "serverless.consul.hashicorp.com/alpha/lambda/invocation-mode" = "ASYNCHRONOUS" + } +} +``` + +### Supported tags + +The following tags are supported. The path prefix for all tags is `serverless.consul.hashicorp.com/v1alpha1/lambda`. For example, specify the following tag to enable the Lambda registrator to sync the Lambda with Consul: + +`serverless.consul.hashicorp.com/v1alpha1/lambda/enabled`. + +| Tag | Description | +| - | - | +| `/enabled` | Enables the Lambda registrator to sync the Lambda with Consul. | +| `/payload-passthrough` | Determines if the body Envoy receives is converted to JSON or directly passed to Lambda. This attribute is optional and defaults to `false`. | +| `/invocation-mode` | Specifies the [Lambda invocation mode](https://docs.aws.amazon.com/lambda/latest/operatorguide/invocation-modes.html) Consul uses to invoke the Lambda. The default is `SYNCHRONOUS`, but `ASYNCHRONOUS` invocations are also supported. | +| `/datacenter` | Specifies the Consul datacenter in which to register the service. The default is the datacenter configured for Lambda registrator. | +| `/namespace` | Specifies the Consul namespace the service will be registered in. Default is `default` if `enterprise` is enabled. | +| `/partition` | Specifies the Consul partition the service will be registered in. Defaults is `default` if `enterprise` is enabled. | +| `/aliases` | Specifies a `+`-separated string of Lambda aliases that will be registered into Consul. For example, if set to `dev+staging+prod`, the `dev`, `staging`, and `prod` aliases of the Lambda function will be registered into Consul. | diff --git a/website/content/docs/lambda/registration/index.mdx b/website/content/docs/lambda/registration/index.mdx new file mode 100644 index 0000000000..eaa797728f --- /dev/null +++ b/website/content/docs/lambda/registration/index.mdx @@ -0,0 +1,78 @@ +--- +layout: docs +page_title: Lambda Function Registration Requirements +description: >- + This topic provides an overview of how to register AWS Lambda functions with Consul service mesh and describes the requirements and prerequisites for registering Lambda functions with Consul. +--- +# Lambda Function Registration Requirements + +Verify that your network meets the requirements and that you have completed the prerequisites before registering Lambda functions. + +## Introduction + +You can either manually register AWS Lambda functions with Consul or use the Lambda registrator to automatically synchronize Lambda state into Consul. We recommend using the Lambda registrator when possible so that you can keep the configuration entry up to date. The registrator automatically registers, reconfigures, and deregisters Lambdas based on the Lambda function's tags. + +## Requirements + +Consul 1.12.1 and later + +## Prerequisites + +Complete the following prerequisites prior to registering your Lambda functions. You only need to perform these steps once. + +### Enable the Serverless Plugin + +Add the following configuration to all Consul clients: + +`connect { enable_serverless_plugin = true, connect = true }` + +Refer to the [`enable_serverless_plugin`](/docs/agent/config/config-files#connect_enable_serverless_plugin) configuration documentation for additional information. + +### Configure IAM Permissions for Envoy + +The Envoy proxy that invokes Lambda must have the `lambda:InvokeFunction` AWS IAM +permissions. In the following example, the IAM policy +enables an IAM user or role to invoke the `example` Lambda function: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Invoke", + "Effect": "Allow", + "Action": [ + "lambda:InvokeFunction" + ], + "Resource": "arn:aws:lambda:us-east-1:123456789012:function:example" + } + ] +} +``` + +Define AWS IAM credentials in environment variables, EC2 metadata or +ECS metadata. On [AWS EKS](https://aws.amazon.com/eks/), associate an IAM role with the proxy's `ServiceAccount`. Refer to the [AWS IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) documentation for instructions. + +### Optional: Set up a Terminating Gateway + +If you intend to invoke Lambda services through a terminating gateway, the gateway must be registered and running in the Consul datacenter. Refer to the following documentation and tutorials for instructions on how to set up a terminating gateway: + +- [Terminating gateways documentation](/docs/connect/gateways#terminating-gateways) +- [Terminating gateways on Kubernetes documentation](/docs/k8s/connect/terminating-gateways) +- [Connect External Services to Consul With Terminating Gateways tutorial](https://learn.hashicorp.com/tutorials/consul/teminating-gateways-connect-external-services) + +To register a Lambda service with a terminating gateway, add the service to the +`Services` field of the terminating gateway's `terminating-gateway` +configuration entry. + +### Optional: Run a Mesh Gateway + +You can set up a mesh gateway so that you can invoke Lambda services across datacenters and admin partitions. The mesh gateway must be running and registered in the relevant Consul datacenters and partitions. Refer to the following documentation and tutorials for instructions on how to set up mesh gateways: + +- [Mesh gateway documentation](/docs/connect/gateways#mesh-gateways) +- [Connect Services Across Datacenters with Mesh Gateways tutorial](https://learn.hashicorp.com/tutorials/consul/service-mesh-gateways) +- [Secure Service Mesh Communication Across Kubernetes Clusters tutorial](https://learn.hashicorp.com/tutorials/consul/kubernetes-mesh-gateways?utm_source=docs?in=consul/kubernetes) + +When using admin partitions, you must add Lambda services to the `Services` +field of [the `exported-services` configuration +entry](/docs/connect/config-entries/exported-services). \ No newline at end of file diff --git a/website/content/docs/lambda/registration/manual.mdx b/website/content/docs/lambda/registration/manual.mdx new file mode 100644 index 0000000000..01f475ecbc --- /dev/null +++ b/website/content/docs/lambda/registration/manual.mdx @@ -0,0 +1,84 @@ +--- +layout: docs +page_title: Manual Lambda Function Registration +description: >- + Register AWS Lambda functions with Consul service mesh using the Consul Lambda registrator. The Consul Lambda registrator automates Lambda function registration. +--- + +# Manual Lambda Function Registration + +This topic describes how to manually register Lambda functions into Consul. Refer to [Automate Lambda Function Registration](/docs/lambda/registration/automate) for information about using the Lambda registrator to automate registration. + +## Requirements + +Verify that your environment meets the requirements specified in [Lambda Function Registration Requirements](/docs/lambda/registration/index). + +To manually register Lambda functions so that mesh services can invoke them, you must create and apply a service registration configuration for the Lambda function and write a [service defaults configuration entry](/docs/connect/config-entries/service-defaults) for the function. + +## Register a Lambda function + +You can manually register Lambda functions if you are unable to automate the process using the Lambda registrator. + +1. Create a configuration for registering the service. You can copy the following example and replace `` with your Consul service name for the Lambda function: + + + + ```json + { + "Node": "lambdas", + "SkipNodeUpdate": true, + "NodeMeta": { + "external-node": "true", + "external-probe": "true" + }, + "Service": { + "Service": "" + } + } + ``` + + + +1. Save the configuration to `lambda.json`. + +1. Send the configuration to the `catalog/register` API endpoint to register the service, for example: + ```shell-session + $ curl --request PUT --data @lambda.json localhost:8500/v1/catalog/register + ``` + +1. Create the `service-defaults` configuration entry and include the AWS tags used to invoke the Lambda function in the `Meta` field (see [Supported `Meta` fields](#supported-meta-fields). The following example creates a `service-defaults` configuration entry named `lambda`: + + + + ```hcl + Kind = "service-defaults" + Name = "lambda" + Protocol = "http" + Meta = { + "serverless.consul.hashicorp.com/v1alpha1/lambda/enabled" = "true" + "serverless.consul.hashicorp.com/v1alpha1/lambda/arn" = "" + "serverless.consul.hashicorp.com/v1alpha1/lambda/payload-passthrough" = "true" + "serverless.consul.hashicorp.com/v1alpha1/lambda/region" = "us-east-2" + } + ``` + + + +1. Issue the `consul config write` command to store the configuration entry. For example: + ```shell-session + $ consul config write lambda-service-defaults.hcl + ``` + +### Supported `Meta` fields + +The following tags are supported. The path prefix for all tags is `serverless.consul.hashicorp.com/v1alpha1/lambda`. For example, specify the following tag to enable Consul to configure the service as an AWS Lambda function: + +`serverless.consul.hashicorp.com/v1alpha1/lambda/enabled`. + +| Tag | Description | +| --- | --- | +| `/enabled` | Determines if Consul configures the service as an AWS Lambda. | +| `/payload-passthrough` | Determines if the body Envoy receives is converted to JSON or directly passed to Lambda. | +| `/arn` | Specifies the [AWS ARN](https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) for the service's Lambda. | +| `/invocation-mode` | Determines if Consul configures the Lambda to be invoked using the `synchronous` or `asynchronous` [invocation mode](https://docs.aws.amazon.com/lambda/latest/operatorguide/invocation-modes.html). | +| `/region` | Specifies the AWS region the Lambda is running in. | diff --git a/website/data/docs-nav-data.json b/website/data/docs-nav-data.json index 6b94e594bd..955f51cb5e 100644 --- a/website/data/docs-nav-data.json +++ b/website/data/docs-nav-data.json @@ -1095,11 +1095,28 @@ }, { "title": "Register Lambda Functions", - "path": "lambda/registration" + "routes":[ + { + "title": "Requirements", + "path": "lambda/registration" + }, + { + "title": "Automate Registration", + "path": "lambda/registration/automate" + }, + { + "title": "Manual Registration", + "path": "lambda/registration/manual" + } + ] }, { - "title": "Invoke Lambda Functions", + "title": "Invoke Lambda Functions from Services", "path": "lambda/invocation" + }, + { + "title": "Invoke Services from Lambda Functions", + "path": "lambda/invoke-from-lambda" } ] }, diff --git a/website/public/img/invoke-service-from-lambda-flow.svg b/website/public/img/invoke-service-from-lambda-flow.svg new file mode 100644 index 0000000000..b67e54c6e7 --- /dev/null +++ b/website/public/img/invoke-service-from-lambda-flow.svg @@ -0,0 +1 @@ + \ No newline at end of file