mirror of
https://github.com/status-im/consul.git
synced 2025-01-29 15:05:04 +00:00
CE-657 - Move Application leader election tutorial to docs (#21366)
* First commit * Fix navigation * Add some commands * Structure draft * Complete usage doc structure * Fix link * Apply suggestions from code review Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> * Apply suggestions from code review * Replace tutorial path * Apply suggestions from code review Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> --------- Co-authored-by: Aimee Ukasick <aimee.ukasick@hashicorp.com> Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com> Co-authored-by: boruszak <jeffrey.boruszak@hashicorp.com>
This commit is contained in:
danielehc
GitHub
parent
58fad92cd3
commit
e2bb1b76cc
@ -212,7 +212,7 @@ The corresponding CLI command is [`consul kv put`](/consul/commands/kv/put).
|
||||
session has locked the key.**
|
||||
|
||||
For an example of how to use the lock feature, check the
|
||||
[Leader Election tutorial](/consul/tutorials/developer-configuration/application-leader-elections).
|
||||
[Leader Election tutorial](/consul/docs/dynamic-app-config/sessions/application-leader-election).
|
||||
|
||||
- `release` `(string: "")` - Supply a session ID to use in a release operation. This is
|
||||
useful when paired with `?acquire=` as it allows clients to yield a lock. This
|
||||
|
||||
@ -77,7 +77,7 @@ The table below shows this endpoint's support for
|
||||
86400s). If provided, the session is invalidated if it is not renewed before
|
||||
the TTL expires. The lowest practical TTL should be used to keep the number of
|
||||
managed sessions low. When locks are forcibly expired, such as when following
|
||||
the [leader election pattern](/consul/tutorials/developer-configuration/application-leader-elections) in an application,
|
||||
the [leader election pattern](/consul/docs/dynamic-app-config/sessions/application-leader-election) in an application,
|
||||
sessions may not be reaped for up to double this TTL, so long TTL
|
||||
values (> 1 hour) should be avoided. Valid time units include "s", "m" and "h".
|
||||
|
||||
|
||||
@ -18,7 +18,7 @@ communication is disrupted, the child process is terminated.
|
||||
|
||||
The number of lock holders is configurable with the `-n` flag. By default,
|
||||
a single holder is allowed, and a lock is used for mutual exclusion. This
|
||||
uses the [leader election algorithm](/consul/tutorials/developer-configuration/application-leader-elections).
|
||||
uses the [leader election algorithm](/consul/docs/dynamic-app-config/sessions/application-leader-election).
|
||||
|
||||
If the lock holder count is more than one, then a semaphore is used instead.
|
||||
A semaphore allows more than a single holder, but this is less efficient than
|
||||
|
||||
@ -111,7 +111,7 @@ increment to the `LockIndex` and the session value is updated to reflect the
|
||||
session holding the lock. Review the session documentation for more information
|
||||
on the [integration](/consul/docs/dynamic-app-config/sessions#k-v-integration).
|
||||
|
||||
Review the following tutorials to learn how to use Consul sessions for [application leader election](/consul/tutorials/developer-configuration/application-leader-elections) and
|
||||
Review the following tutorials to learn how to use Consul sessions for [application leader election](/consul/docs/dynamic-app-config/sessions/application-leader-election) and
|
||||
to [build distributed semaphores](/consul/tutorials/developer-configuration/distributed-semaphore).
|
||||
|
||||
### Vault
|
||||
|
||||
@ -0,0 +1,396 @@
|
||||
---
|
||||
layout: docs
|
||||
page_title: Application leader election
|
||||
description: >-
|
||||
Learn how to perform client-side leader elections using sessions and Consul key/value (KV) store.
|
||||
---
|
||||
|
||||
# Application leader election
|
||||
|
||||
This topic describes the process for building client-side leader elections for service instances using Consul's [session mechanism for building distributed locks](/consul/docs/dynamic-app-config/sessions) and the [Consul key/value store](/consul/docs/dynamic-app-config/kv), which is Consul's key/value datastore.
|
||||
|
||||
This topic is not related to Consul's leader election. For more information about the Raft leader election used internally by Consul, refer to
|
||||
[consensus protocol](/consul/docs/architecture/consensus) documentation.
|
||||
|
||||
## Background
|
||||
|
||||
Some distributed applications, like HDFS or ActiveMQ, require setting up one instance as a leader to ensure application data is current and stable.
|
||||
|
||||
Consul's support for [sessions](/consul/docs/dynamic-app-config/sessions) and [watches](/consul/docs/dynamic-app-config/watches) allows you to build a client-side leader election process where clients use a lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures.
|
||||
|
||||
All service instances that are participating should coordinate on a key format. We recommend the following pattern:
|
||||
|
||||
```plaintext
|
||||
service/<service name>/leader
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- A running Consul server
|
||||
- A path in the Consul KV datastore to acquire locks and to store information about the leader. The instructions on this page use the following key: `service/leader`.
|
||||
- If ACLs are enabled, a token with the following permissions:
|
||||
- `session:write` permissions over the service session name
|
||||
- `key:write` permissions over the key
|
||||
- The `curl` command
|
||||
|
||||
Expose the token using the `CONSUL_HTTP_TOKEN` environment variable.
|
||||
|
||||
## Client-side leader election procedure
|
||||
|
||||
The workflow for building a client-side leader election process has the following steps:
|
||||
|
||||
- For each client trying to acquire the lock:
|
||||
1. [Create a session](#create-a-new-session) associated with the client node.
|
||||
1. [Acquire the lock](#acquire-the-lock) on the designated key in the KV store using the `acquire` parameter.
|
||||
1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify if the lock was released. If no lock is present, try to acquire a lock.
|
||||
|
||||
- For the client that acquires the lock:
|
||||
1. Periodically, [renew the session](#renew-a-session) to avoid expiration.
|
||||
1. Optionally, [release the lock](#release-a-lock).
|
||||
|
||||
- For other services:
|
||||
1. [Watch the KV key](#watch-the-kv-key-for-locks) to verify there is at least one process holding the lock.
|
||||
1. Use the values written under the KV path to identify the leader and update configurations accordingly.
|
||||
|
||||
## Create a new session
|
||||
|
||||
Create a configuration for the session.
|
||||
The minimum viable configuration requires that you specify the session name. The following example demonstrates this configuration.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```json
|
||||
{
|
||||
"Name": "session_name"
|
||||
}
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
Create a session using the [`/session` Consul HTTP API](/consul/api-docs/session) endpoint. In the following example, the node's `hostname` is the session name.
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--data '{"Name": "'`hostname`'"}' \
|
||||
--request PUT \
|
||||
http://127.0.0.1:8500/v1/session/create | jq
|
||||
```
|
||||
|
||||
|
||||
The command returns a JSON object containing the ID of the newly created session.
|
||||
|
||||
```json
|
||||
{
|
||||
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6"
|
||||
}
|
||||
```
|
||||
|
||||
### Verify session
|
||||
|
||||
Use the `/v1/session/list` endpoint to retrieve existing sessions.
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--request GET \
|
||||
http://127.0.0.1:8500/v1/session/list | jq
|
||||
```
|
||||
|
||||
The command returns a JSON array containing all available sessions in the system.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
|
||||
"Name": "hashicups-db-0",
|
||||
"Node": "hashicups-db-0",
|
||||
"LockDelay": 15000000000,
|
||||
"Behavior": "release",
|
||||
"TTL": "",
|
||||
"NodeChecks": [
|
||||
"serfHealth"
|
||||
],
|
||||
"ServiceChecks": null,
|
||||
"CreateIndex": 11956,
|
||||
"ModifyIndex": 11956
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
You can verify from the output that the session is associated with the `hashicups-db-0` node, which is the client agent where the API request was made.
|
||||
|
||||
With the exception of the `Name`, all parameters are set to their default values. The session is created without a `TTL` value, which means that it never expires and requires you to delete it explicitly.
|
||||
|
||||
Depending on your needs you can create sessions specifying more parameters such as:
|
||||
|
||||
- `TTL` - If provided, the session is invalidated and deleted if it is not renewed before the TTL expires.
|
||||
- `ServiceChecks` - Specifies a list of service checks to monitor. The session is invalidated if the checks return a critical state.
|
||||
|
||||
By setting these extra parameters, you can create a client-side leader election workflow that automatically releases the lock after a specified amount of time since the last renew, or that automatically releases locks when the service holding them fails.
|
||||
|
||||
For a full list of parameters available refer to the [`/session/create` endpoint documentation](/consul/api-docs/session#create-session).
|
||||
|
||||
## Acquire the lock
|
||||
|
||||
Create the data object to associate to the lock request.
|
||||
|
||||
The data of the request should be a JSON object representing the local instance. This value is opaque to Consul, but it should contain whatever information clients require to communicate with your application. For example, it could be a JSON object that contains the node's name and the application's port.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```json
|
||||
{
|
||||
"Node": "node-name",
|
||||
"Port": "8080"
|
||||
}
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
<Tabs>
|
||||
<Tab heading="API" group="api">
|
||||
|
||||
Acquire a lock for a given key using the PUT method on a [KV entry](/consul/api-docs/kv) with the
|
||||
`?acquire=<session>` query parameter.
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--data '{"Node": "'`hostname`'"}' \
|
||||
--request PUT \
|
||||
http://localhost:8500/v1/kv/service/leader?acquire=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq
|
||||
```
|
||||
|
||||
This request returns either `true` or `false`. If `true`, the lock was acquired and
|
||||
the local service instance is now the leader. If `false`, a different node acquired
|
||||
the lock.
|
||||
|
||||
|
||||
</Tab>
|
||||
<Tab heading="CLI" group="cli">
|
||||
|
||||
```shell-session
|
||||
$ consul kv put -acquire -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 /service/leader '{"Node": "'`hostname`'"}'
|
||||
```
|
||||
|
||||
In case of success, the command exits with exit code `0` and outputs the following message.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```plaintext
|
||||
Success! Lock acquired on: service/leader
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
If the lock was already acquired by another node, the command exits with exit code `1` and outputs the following message.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```plaintext
|
||||
Error! Did not acquire lock
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
This example used the node's `hostname` as the key data. This data can be used by the other services to create configuration files.
|
||||
|
||||
Be aware that this locking system has no enforcement mechanism that requires clients to acquire a lock before they perform an operation. Any client can read, write, and delete a key without owning the corresponding lock.
|
||||
|
||||
## Watch the KV key for locks
|
||||
|
||||
Existing locks need to be monitored by all nodes involved in the client-side leader elections, as well as by the other nodes that need to know the identity of the leader.
|
||||
|
||||
- Lock holders need to monitor the lock because the session might get invalidated by an operator.
|
||||
- Other services that want to acquire the lock need to monitor it to check if the lock is released so they can try acquire the lock.
|
||||
- Other nodes need to monitor the lock to see if the value of the key changed and update their configuration accordingly.
|
||||
|
||||
<Tabs>
|
||||
<Tab heading="API" group="api">
|
||||
|
||||
Monitor the lock using the GET method on a [KV entry](/consul/api-docs/kv) with the blocking query enabled.
|
||||
|
||||
First, verify the latest index for the current value.
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--request GET \
|
||||
http://127.0.0.1:8500/v1/kv/service/leader?index=1 | jq
|
||||
```
|
||||
|
||||
The command outputs the key data, including the `ModifyIndex` for the object.
|
||||
|
||||
<CodeBlockConfig hideClipboard highlight="9">
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"LockIndex": 0,
|
||||
"Key": "service/leader",
|
||||
"Flags": 0,
|
||||
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=",
|
||||
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
|
||||
"CreateIndex": 12399,
|
||||
"ModifyIndex": 13061
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
Using the value of the `ModifyIndex`, run a [blocking query](/consul/api-docs/features/blocking) against the lock.
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--request GET \
|
||||
http://127.0.0.1:8500/v1/kv/service/leader?index=13061 | jq
|
||||
```
|
||||
The command hangs until a change is made on the KV path and after that the path data prints on the console.
|
||||
|
||||
<CodeBlockConfig hideClipboard highlight="9">
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"LockIndex": 0,
|
||||
"Key": "service/leader",
|
||||
"Flags": 0,
|
||||
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0xIn0=",
|
||||
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
|
||||
"CreateIndex": 12399,
|
||||
"ModifyIndex": 13329
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
For automation purposes, add logic to the blocking query mechanism to trigger a command every time a change is returned.
|
||||
A better approach is to use the CLI command `consul watch`.
|
||||
|
||||
</Tab>
|
||||
<Tab heading="CLI" group="cli">
|
||||
|
||||
Monitor the lock using the [`consul watch`](/consul/commands/watch) command.
|
||||
|
||||
```shell-session
|
||||
$ consul watch -type=key -key=service/leader cat | jq
|
||||
```
|
||||
|
||||
In this example, the command output prints to the shell. However, it is possible to pass more complex option to the command as well as a script that contains more complex logic to react to the lock data change.
|
||||
|
||||
An example output for the command is:
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```json
|
||||
{
|
||||
"Key": "service/leader",
|
||||
"CreateIndex": 12399,
|
||||
"ModifyIndex": 13061,
|
||||
"LockIndex": 0,
|
||||
"Flags": 0,
|
||||
"Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=",
|
||||
"Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6"
|
||||
}
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
The `consul watch` command polls the KV path for changes and runs the specified command on the output when a change is made.
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
From the output, notice that once the lock is acquired, the `Session` parameter contains the ID of the session that holds the lock.
|
||||
|
||||
## Renew a session
|
||||
|
||||
If a session is created with a `TTL` value set, you need to renew the session before the TTL expires.
|
||||
|
||||
Use the [`/v1/session/renew`](/consul/api-docs/session#renew-session) endpoint to renew existing sessions.
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--request PUT \
|
||||
http://127.0.0.1:8500/v1/session/renew/f027470f-2759-6b53-542d-066ae4185e67 | jq
|
||||
```
|
||||
|
||||
If the command succeeds, the session information in JSON format is printed.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"ID": "f027470f-2759-6b53-542d-066ae4185e67",
|
||||
"Name": "test",
|
||||
"Node": "consul-server-0",
|
||||
"LockDelay": 15000000000,
|
||||
"Behavior": "release",
|
||||
"TTL": "30s",
|
||||
"NodeChecks": [
|
||||
"serfHealth"
|
||||
],
|
||||
"ServiceChecks": null,
|
||||
"CreateIndex": 11842,
|
||||
"ModifyIndex": 11842
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
## Release a lock
|
||||
|
||||
A lock associated with a session with no `TTL` value set might never be released, even when the service holding it fails.
|
||||
|
||||
In such cases, you need to manually release the lock.
|
||||
|
||||
<Tabs>
|
||||
<Tab heading="API" group="api">
|
||||
|
||||
```shell-session
|
||||
$ curl --silent \
|
||||
--header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
|
||||
--data '{"Node": "'`hostname`'"}' \
|
||||
--request PUT \
|
||||
http://localhost:8500/v1/kv/service/leader?release=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq
|
||||
```
|
||||
|
||||
The command prints `true` on success.
|
||||
|
||||
</Tab>
|
||||
<Tab heading="CLI" group="cli">
|
||||
|
||||
```shell-session
|
||||
$ consul kv put -release -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 service/leader '{"Node": "'`hostname`'"}'
|
||||
```
|
||||
|
||||
On success, the command outputs a success message.
|
||||
|
||||
<CodeBlockConfig hideClipboard>
|
||||
|
||||
```plaintext
|
||||
Success! Lock released on: service/leader
|
||||
```
|
||||
|
||||
</CodeBlockConfig>
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
After a lock is released, the key data do not show a value for `Session` in the results.
|
||||
Other clients can use this as a way to coordinate their lock requests.
|
||||
|
||||
@ -134,9 +134,9 @@ the goal of Consul to protect against misbehaving clients.
|
||||
|
||||
## Leader Election
|
||||
|
||||
The primitives provided by sessions and the locking mechanisms of the KV
|
||||
store can be used to build client-side leader election algorithms.
|
||||
These are covered in more detail in the [Leader Election guide](/consul/tutorials/developer-configuration/application-leader-elections).
|
||||
You can use the primitives provided by sessions and the locking mechanisms of the KV
|
||||
store to build client-side leader election algorithms.
|
||||
These are covered in more detail in the [Leader Election guide](/consul/docs/dynamic-app-config/sessions/application-leader-election).
|
||||
|
||||
## Prepared Query Integration
|
||||
|
||||
@ -1054,7 +1054,16 @@
|
||||
},
|
||||
{
|
||||
"title": "Sessions",
|
||||
"path": "dynamic-app-config/sessions"
|
||||
"routes": [
|
||||
{
|
||||
"title": "Overview",
|
||||
"path": "dynamic-app-config/sessions"
|
||||
},
|
||||
{
|
||||
"title": "Application leader election",
|
||||
"path": "dynamic-app-config/sessions/application-leader-election"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Watches",
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user