Commit Graph

101 Commits

Author SHA1 Message Date
James Phillips c01a3871c9 Adds support for snapshots and restores. (#2396)
* Updates Raft library to get new snapshot/restore API.

* Basic backup and restore working, but need some cleanup.

* Breaks out a snapshot module and adds a SHA256 integrity check.

* Adds snapshot ACL and fills in some missing comments.

* Require a consistent read for snapshots.

* Make sure snapshot works if ACLs aren't enabled.

* Adds a bit of package documentation.

* Returns an empty response from restore to avoid EOF errors.

* Adds API client support for snapshots.

* Makes internal file names match on-disk file snapshots.

* Adds DC and token coverage for snapshot API test.

* Adds missing documentation.

* Adds a unit test for the snapshot client endpoint.

* Moves the connection pool out of the client for easier testing.

* Fixes an incidental issue in the prepared query unit test.

I realized I had two servers in bootstrap mode so this wasn't a good setup.

* Adds a half close to the TCP stream and fixes panic on error.

* Adds client and endpoint tests for snapshots.

* Moves the pool back into the snapshot RPC client.

* Adds a TLS test and fixes half-closes for TLS connections.

* Tweaks some comments.

* Adds a low-level snapshot test.

This is independent of Consul so we can pull this out into a library
later if we want to.

* Cleans up snapshot and archive and completes archive tests.

* Sends a clear error for snapshot operations in dev mode.

Snapshots require the Raft snapshots to be readable, which isn't supported
in dev mode. Send a clear error instead of a deep-down Raft one.

* Adds docs for the snapshot endpoint.

* Adds a stale mode and index feedback for snapshot saves.

This gives folks a way to extract data even if the cluster has no
leader.

* Changes the internal format of a snapshot from zip to tgz.

* Pulls in Raft fix to cancel inflight before a restore.

* Pulls in new Raft restore interface.

* Adds metadata to snapshot saves and a verify function.

* Adds basic save and restore snapshot CLI commands.

* Gets rid of tarball extensions and adds restore message.

* Fixes an incidental bad link in the KV docs.

* Adds documentation for the snapshot CLI commands.

* Scuttle any request body when a snapshot is saved.

* Fixes archive unit test error message check.

* Allows for nil output writers in snapshot RPC handlers.

* Renames hash list Decode to DecodeAndVerify.

* Closes the client connection for snapshot ops.

* Lowers timeout for restore ops.

* Updates Raft vendor to get new Restore signature and integrates with Consul.

* Bounces the leader's internal state when we do a restore.
2016-10-25 19:20:24 -07:00
Felix f0eb0ba549 return an error if no logOutput is provided
Otherwise the code will panic at a later stage where it is more complicated to figure out what’s actually going on.
2016-09-17 17:46:13 +09:00
James Phillips e5850d8a26
Adds new consul operator endpoint, CLI, and ACL and some basic Raft commands. 2016-08-30 00:02:50 -07:00
James Phillips 9f7a973ace
Adds an `X-Consul-Translate-Addresses` to signal translation is enabled. 2016-08-16 11:31:41 -07:00
James Phillips f73166ced9
Installs a wrapper around HTTP endpoints that generate metrics. 2016-08-09 15:41:15 -07:00
James Phillips d29af2ddc7
Adds an ACL replication status endpoint. 2016-08-04 23:30:16 -07:00
James Phillips 69f58ad04a Moves txn code into a new endpoint, not specific to KV. 2016-05-10 21:58:02 -07:00
James Phillips 1fefdcb962 Terminates pretty responses with a newline. 2016-05-10 20:02:36 -07:00
James Phillips e491245062 Performs basic plumbing of KVS transactions through all the layers. 2016-05-09 22:15:49 -07:00
James Phillips 7ebad899da Adds a new PUT-based TTL check update endpoint. 2016-03-02 17:54:01 -08:00
James Phillips e79dd7c8de Fixes redirect from / to /ui when internal UI is enabled. 2016-02-12 16:11:32 -08:00
Chris Cooper e2065e10c0 add comment codes 2016-02-05 17:06:42 -05:00
Chris Cooper 00dd9d6b78 refactors http server error codes to use the http statuses defined in the http package instead of literals 2016-02-05 16:51:31 -05:00
Philippe M. Chiasson a2eddcab38 remove address splitting 2016-01-05 11:49:40 -05:00
Philippe M. Chiasson 21345a8a7f Move logged client IP to the end of the log message 2015-12-30 15:39:34 -05:00
Philippe M. Chiasson 34c36d9205 Merge branch 'master' of http://github.com/hashicorp/consul into issue/1447/http-access-logs 2015-12-30 15:38:30 -05:00
Ryan Uber 3cf6a173a5 agent: static UI is configurable 2015-12-24 22:21:34 -05:00
Ryan Uber c3e7c98da2 agent: compile web assets into consul binary 2015-12-24 22:21:34 -05:00
Philippe M. Chiasson 65512cae73 Add client ip to http access logs
Fixes #1447
2015-11-24 15:06:33 -05:00
James Phillips da20e6668b Adds a note about obfuscating query name/ID from the logs. 2015-11-15 17:06:00 -08:00
James Phillips 57be55103c Adds an HTTP endpoint for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips d45fc23abf Installs a friendly handler for coordinate endpoints when coordinates are disabled. 2015-10-23 15:23:01 -07:00
James Phillips 33e3505aea Adds endpoints for raw network coordinates. 2015-10-23 15:23:01 -07:00
James Phillips 9c9195746f Changes ?near=self to a safer ?near=_agent, which is also clearer about what it does. 2015-10-23 15:23:01 -07:00
James Phillips e47eea3f3a Adds a magic "self" node name to distance queries. 2015-10-23 15:23:01 -07:00
James Phillips 89c7203f31 Adds coordinate sorting support to catalog queries for nodes and service nodes. 2015-10-23 15:23:01 -07:00
David Adams b7bcb2a414 Add HTTP request header X-Consul-Token
Add support for an X-Consul-Token HTTP request header to specify the
token with which this request should be fulfilled. The header would have
precedence over the responding Agent's default token, but would have
lower precedence than a token specified in the query string.
2015-10-19 11:26:01 -05:00
Michael Puncel a98d25b541 Add http method to log output 2015-10-02 18:33:06 -07:00
Armon Dadgar 52a8a95af9 agent: Always enable the UI endpoints 2015-09-01 18:28:32 -07:00
Ryan Uber 87c1e4fcd3 agent: document the scada http creation func 2015-08-25 17:19:11 -07:00
Ryan Uber 2e6ccded2c agent: scada client and HTTP server are tracked separately 2015-08-25 16:59:53 -07:00
Ryan Uber 1b4167699f agent: don't replace empty tokens in the logs, fixes #1020 2015-06-12 00:11:37 -07:00
Ryan Uber 507917748a agent: parse raw query URL to avoid closing the request body early 2015-04-13 17:31:53 -07:00
Ryan Uber ee5659858a agent: hide tokens from logs and monitor 2015-04-12 11:17:31 -07:00
Armon Dadgar 847eaadc56 agent: Enable UI endpoints for SCADA 2015-02-18 15:15:02 -08:00
Armon Dadgar c495a5434d agent: Use AtlasACLToken 2015-02-18 15:12:15 -08:00
Armon Dadgar b9cdb94f19 agent: Fixing setup of SCADA HTTP listener 2015-02-18 15:12:15 -08:00
Armon Dadgar 6a640604dd agent: Fixing panic on shutdown 2015-02-18 15:12:15 -08:00
Armon Dadgar 456645f2fb agent: SCADA HTTP listener 2015-02-18 15:12:14 -08:00
Ryan Uber b5e8111485 agent/http: clean up socket setup 2015-01-23 17:57:04 -08:00
Armon Dadgar cf04d6ae31 Merge pull request #622 from hashicorp/f-sockets
Unix domain sockets
2015-01-21 16:30:03 -08:00
Ryan Uber d9a3e673b0 agent: change node maintenance endpoint 2015-01-21 10:57:44 -08:00
Ryan Uber 21dd95d9af agent: beginning socket user/group/mode support as discussed in #612 2015-01-20 13:44:27 -08:00
Ryan Uber 7748c62d09 agent: node maintenance mode works 2015-01-16 15:38:13 -08:00
Ryan Uber 2973cd9131 agent: first pass at service maintenance mode 2015-01-16 15:37:51 -08:00
Ryan Uber f01bb5cf3b agent: error if binding to existing socket file 2015-01-16 12:39:15 -08:00
Ryan Uber bf48651c58 agent: only ignore errors on IsNotExist() 2015-01-16 09:14:52 -08:00
Ryan Uber 4675cdf01c agent: beginning refactor 2015-01-16 00:45:03 -08:00
Jeff Mitchell 11a3ce0bdd RPC and HTTP interfaces fully generically-sockified so Unix is supported.
Client works for RPC; will honor CONSUL_RPC_ADDR. HTTP works via consul/api;
honors CONSUL_HTTP_ADDR.

The format of a Unix socket in configuration data is:
"unix://[/path/to/socket];[username or uid];[gid];[mode]"

Obviously, the user must have appropriate permissions to create the socket
file in the given path and assign the requested uid/gid. Also note that Go does
not support gid lookups from group name, so gid must be numeric. See
https://codereview.appspot.com/101310044

When connecting from the client, the format is just the first part of the
above line:
"unix://[/path/to/socket]"

This code is copyright 2014 Akamai Technologies, Inc. <opensource@akamai.com>
2015-01-14 19:31:21 +00:00
Armon Dadgar f86d7c3a09 Merge pull request #558 from ceh/http-api-response-headers
add ability to specify response headers on the HTTP API
2015-01-05 11:36:08 -08:00