Commit Graph

666 Commits

Author SHA1 Message Date
James Phillips 1c6de1d623
Fixes all the racy output-side updates to tags. 2018-02-06 20:35:55 -08:00
James Phillips 11f6961e47
Adds a more robust unit test for index churn. 2018-02-06 20:35:38 -08:00
Pierre Souchay 80dde5465b Added support for Service Metadata 2018-02-07 01:54:42 +01:00
James Phillips d9a6e2a901
Makes server manager shift away from failed servers from Serf events.
Because this code was doing pointer equality checks, it would work for
the case of a failed attempted RPC because the objects are from the
manager itself:

https://github.com/hashicorp/consul/blob/v1.0.3/agent/consul/rpc.go#L283-L302

But the pointer check would always fail for events coming in from the
Serf path because the server object is newly-created:

https://github.com/hashicorp/consul/blob/v1.0.3/agent/router/serf_adapter.go#L14-L40

This means that we didn't proactively shift RPC traffic away from a
failed server, we'd have to wait for an RPC to fail, which exposes
the error to the calling client.

By switching over to a name check vs. a pointer check we get the correct
behavior. We added a DEBUG log as well to help observe this behavior during
integrated testing.

Related to #3863 since the fix here needed the same logic duplicated, owing
to the complicated atomic stuff.

/cc @dadgar for a heads up in case this also affects Nomad.
2018-02-05 17:56:00 -08:00
James Phillips fc155dac19
Adds a before/after test for #3845. 2018-02-05 16:18:29 -08:00
James Phillips 533f65b7a6
Merge pull request #3845 from 42wim/tagfix
Fix service tags not added to health check. Part two
2018-02-05 16:18:00 -08:00
Kyle Havlovitz f6ecaa4a1c
Add enterprise default config section 2018-02-05 13:33:59 -08:00
James Phillips e748c63fff
Merge pull request #3855 from hashicorp/pr-3782-slackpad
Adds support for gRPC health checks.
2018-02-02 17:57:27 -08:00
James Phillips 5f31c8d8d3
Changes "TLS" to "GRPCUseTLS" since it only applies to GRPC checks. 2018-02-02 17:29:34 -08:00
Wim ce771f1fb3 Fix service tags not added to health check. Part two 2018-01-29 20:32:44 +01:00
Veselkov Konstantin 5f38e1148a fix refactoring 2018-01-28 22:53:30 +04:00
Veselkov Konstantin 8e16bd7d77 fix refactoring 2018-01-28 22:48:21 +04:00
Veselkov Konstantin 7de57ba4de remove golint warnings 2018-01-28 22:40:13 +04:00
James Phillips 9cd602de06
Improves user lookup error message.
Closes #3188
Closes #3184
2018-01-26 07:56:44 -08:00
Kyle Havlovitz 144e6e7d31
Remove nonvoter from metadata.Server 2018-01-25 17:08:03 -08:00
James Phillips 64acd0ade0
Gets rid of named return parameters.
This wasn't wrong before but we don't generally use this style in
Consul.
2018-01-25 14:29:50 -08:00
James Phillips b443bd1438
Moves non-stdlib includes into their own section. 2018-01-25 14:26:15 -08:00
Kyle Havlovitz bfeb09983b
Reset clusterHealth when autopilot starts 2018-01-23 12:52:28 -08:00
Kyle Havlovitz 17805e4634
Move autopilot health loop into leader operations 2018-01-23 11:17:41 -08:00
James Phillips c190b35b0e
Updates web assets to latest. 2018-01-22 14:46:07 -08:00
Kyle Havlovitz cde1e7ceb6
Merge pull request #3821 from hashicorp/persist-file-handling
Add graceful handling of malformed persisted service/check files.
2018-01-22 12:31:33 -08:00
Kyle Havlovitz f156b12b22
Merge pull request #3820 from hashicorp/serfwan-port-fix
Enforce a valid port for the Serf WAN since it can't be disabled.
2018-01-19 15:40:56 -08:00
James Phillips 93fd6bfeb4
Moves the coordinate fetch after the ACL check. 2018-01-19 15:25:22 -08:00
Kyle Havlovitz 68ae92cb8c
Don't remove the files, just log an error 2018-01-19 14:25:51 -08:00
Kyle Havlovitz 8c5be2dd97
Enforce a valid port for the Serf WAN since it can't be disabled.
Fixes #3817
2018-01-19 14:22:23 -08:00
Kyle Havlovitz 4e325a6b8f
Add graceful handling of malformed persisted service/check files.
Previously a change was made to make the file writing atomic,
but that wasn't enough to cover something like an OS crash so we
needed something here to handle the situation more gracefully.

Fixes #1221.
2018-01-19 14:07:36 -08:00
James Hartig aedab91a66 Resolve symlinks in config directory
Docker/Openshift/Kubernetes mount the config file as a symbolic link and
IsDir returns true if the file is a symlink. Before calling IsDir, the
symlink should be resolved to determine if it points at a file or
directory.

Fixes #3753
2018-01-12 15:43:38 -05:00
James Phillips 9509aa6c4b
Adds the NodeID field back to the /v1/agent/self Config block.
Fixes #3778
2018-01-10 15:17:54 -08:00
James Phillips ebcd1787db
Adds more info about how to fix the private IP error.
Closes #3790
2018-01-10 09:53:41 -08:00
James Phillips 48cfe6ff5f
Fixes crash where body was optional for PQ endpoint (it is not).
Fixes #3791
2018-01-10 09:33:49 -08:00
Dmytro Kostiuchenko 1a10b08e82 Add gRPC health-check #3073 2018-01-04 16:42:30 -05:00
Diptanu Choudhury 294151c1ad Using labels 2017-12-21 20:30:29 -08:00
Diptanu Choudhury 006eab2394 Added telemetry around Catalog APIs 2017-12-21 16:35:12 -08:00
James Phillips 5b88b8df38
Updates the checked in web assets. 2017-12-20 19:51:04 -08:00
James Phillips 6412d8d9aa
Updates the built-in web assets. 2017-12-20 17:48:51 -08:00
James Phillips 7a46d9c1e3
Wraps HTTP mux to ban all non-printable characters from paths. 2017-12-20 15:47:53 -08:00
James Phillips 2edc11b44c
Updates the built-in web UI assets. 2017-12-20 13:43:52 -08:00
James Phillips da6a4635b0
Fixes a `go fmt` cleanup. 2017-12-20 13:43:38 -08:00
Kyle Havlovitz 11a0c9cc58
Fix vet error 2017-12-18 18:04:42 -08:00
Kyle Havlovitz 77dc52f430
Move autopilot initializing to oss file 2017-12-18 18:02:44 -08:00
Kyle Havlovitz 039e7f1880
Move autopilot setup to a separate file 2017-12-18 16:55:51 -08:00
Kyle Havlovitz d08ab9fd19
Make some final tweaks to autopilot package 2017-12-18 12:26:47 -08:00
Kyle Havlovitz a86d11ec0a
Merge pull request #3737 from hashicorp/autopilot-refactor
Move autopilot to a standalone package
2017-12-15 14:09:40 -08:00
James Phillips 06f980061e
Merge pull request #3728 from weiwei04/fix_globalRPC_goroutine_leak
fix globalRPC goroutine leak
2017-12-14 17:54:19 -08:00
James Phillips f491a55e47
Merge pull request #3642 from yfouquet/master
[Fix] Service tags not added to health checks
2017-12-14 13:59:39 -08:00
James Phillips ca3f9024ac
Works around mapstructure behavior to enable sessions with no checks.
Fixes #3732
2017-12-14 09:07:56 -08:00
Kyle Havlovitz 324c2ecb53
Expose IsPotentialVoter for advanced autopilot logic 2017-12-13 17:53:51 -08:00
James Phillips 98e837167e
Changes maps to merge vs. overwrite when processing configs.
Fixes #3716
2017-12-13 16:06:01 -08:00
Kyle Havlovitz 12bf61c851
Merge branch 'master' into autopilot-refactor 2017-12-13 11:54:32 -08:00
Kyle Havlovitz d6b266c045
A few last autopilot adjustments 2017-12-13 11:19:17 -08:00
Kyle Havlovitz 2310687c1d
More autopilot reorganizing 2017-12-13 10:57:37 -08:00
James Phillips 46742a5041
Adds TODOs referencing #3744. 2017-12-13 10:52:06 -08:00
James Phillips 2892f91d0b
Copies the autopilot settings from the runtime config.
Fixes #3730
2017-12-13 10:32:05 -08:00
Kyle Havlovitz b92f895c23
More refactoring to make autopilot consul-agnostic 2017-12-12 17:46:28 -08:00
Yoann Fouquet 986148cfe5 [Fix] Service tags not added to health checks
Since commit 9685bdcd0b, service tags are added to the health checks.
Otherwise, when adding a service, tags are not added to its check.

In updateSyncState, we compare the checks of the local agent with the checks of the catalog.
It appears that the service tags are different (missing in one case), and so the check is synchronized.
That increase the ModifyIndex periodically when nothing changes.

Fixed it by adding serviceTags to the check.

Note that the issue appeared in version 0.8.2.
Looks related to #3259.
2017-12-12 13:39:37 +01:00
Kyle Havlovitz de28555671
Move autopilot to a standalone package 2017-12-11 16:45:33 -08:00
James Phillips d12e81860f
Moves Serf helper into lib to fix import cycle in consul-enterprise. 2017-12-07 16:57:58 -08:00
James Phillips 5065f3d82e
Turns of intent queue warnings and enables dynamic queue sizing. 2017-12-07 16:27:06 -08:00
Wei Wei cc9648c957 fix globalRPC goroutine leak
Signed-off-by: Wei Wei <weiwei.inf@gmail.com>
2017-12-05 11:53:30 +08:00
James Phillips 3e46544085
Creates a registration mechanism for snapshot and restore. 2017-11-29 18:36:53 -08:00
James Phillips f53f521072
Begins split out of snapshots from the main FSM class. 2017-11-29 18:36:53 -08:00
James Phillips c8e763667f
Creates a registration mechanism for FSM commands. 2017-11-29 18:36:53 -08:00
James Phillips 78292662d7
Moves the FSM into its own package.
This will help make it clearer what happens when we add some registration
plumbing for the different operations and snapshots.
2017-11-29 18:36:53 -08:00
James Phillips e810697e06
Resolves an FSM snapshot TODO.
This adds checks for sink write calls before we continue the refactor, which
will resolve the other TODO comment we deleted as part of this change.
2017-11-29 18:36:53 -08:00
James Phillips aa61159b74
Creates a registration mechanism for schemas.
This also splits out the registration into the table-specific source
files.
2017-11-29 18:36:52 -08:00
James Phillips 93ff33b1be
Creates a registration mechanism for RPC endpoints. 2017-11-29 18:36:52 -08:00
James Phillips 68f100c8df
Creates HTTP endpoint registry. 2017-11-29 18:36:52 -08:00
James Phillips 679775418f
Moves coordinate disabled logic down into endpoints.
Similar rationale to the previous change for ACLs.
2017-11-29 18:36:52 -08:00
James Phillips 29367cd5ae
Moves ACL disabled response logic down into endpoints.
This lets us make the registration of endpoints less fancy, on the
road to adding a registration mechanism.
2017-11-29 18:36:52 -08:00
James Phillips 44d824a58f
Renames "segments" to "segment" to be consistent with other files. 2017-11-29 18:36:52 -08:00
James Phillips 8bf1f57737
Renames stubs to be more consistent. 2017-11-29 18:36:52 -08:00
James Phillips 8abd2050fa
Sheds monotonic time info so tombstone GC bins work properly. 2017-11-29 10:34:24 -08:00
James Phillips de57a9ef51
Gives back the lock before writing to the expire channel.
The lock isn't needed after we clean up the expire bin, and as seen
in #3700 we can get into a deadlock waiting to place the expire index
into the channel while holding this lock.

Fixes #3700
2017-11-19 16:24:16 -08:00
James Phillips 34c13925d4
Skips files with unknown extensions when not forcing a format.
Fixes #3685
2017-11-10 18:06:07 -08:00
James Phillips 1e49c157c5
Adds a snapshot agent stub to the config structure.
Fixes #3678
2017-11-10 13:50:45 -08:00
James Phillips 990fbbb86b
Cleans up check logging.
There were places where we still didn't have the script vs. args sorted
correctly so changed all the logging to be just based on check IDs and
also made everything uniform.

Also removed some annoying debug logging, and moved some of the large output
logging to TRACE level.

Closes #3602
2017-11-10 12:48:44 -08:00
James Phillips f19ba41144
Moves the LAN event handler after the router is created.
Fixes #3680
2017-11-10 12:26:48 -08:00
James Phillips 17737ee030
Revert "Adds a small sleep to make sure we are in the next GC bucket." 2017-11-08 22:18:37 -08:00
James Phillips 24475048e2
Adds a sleep to make sure we are in the next GC bucket, ups time.
Fixes #3670
2017-11-08 22:02:40 -08:00
James Phillips c57884fffe
Skips the tombstone GC test in Travis for now.
Related to #3670
2017-11-08 20:14:20 -08:00
James Phillips cd935ebe6c
Adds missing os import. 2017-11-08 20:02:22 -08:00
James Phillips f6b7dcbcf6
Removes bogus getPort() in favor of freeport. 2017-11-08 19:55:50 -08:00
James Phillips 8579225c27
Skips IPv6 test in Travis. 2017-11-08 18:28:45 -08:00
James Phillips 2937656f8e
Adds a longer retry period for the AE deferred output test.
There's some justification in the comments about this and a TODO to
improve this later.

Fixes #3668
2017-11-08 18:10:13 -08:00
James Phillips 7b966e2d26
Tightens timing up and reorders GC test to be less flaky. 2017-11-08 15:09:29 -08:00
James Phillips 7c6ab5e783
Doubles the GC timing. 2017-11-08 15:01:11 -08:00
James Phillips 8de7c77482
Opens up test timing a little more. 2017-11-08 14:01:19 -08:00
James Phillips c46612f691
Shifts off a gran boundary to help make test less flaky. 2017-11-08 13:57:17 -08:00
James Phillips f31856c1b7
Opens up the tombstone GC test timing. 2017-11-08 13:43:39 -08:00
James Phillips 93f68555d0
Adds enable_agent_tls_for_checks configuration option which allows (#3661)
HTTP health checks for services requiring 2-way TLS to be checked
using the agent's credentials.
2017-11-07 18:22:09 -08:00
James Phillips 85e678fbdd
Saves the cycled server list after a failed ping when rebalancing. (#3662)
Fixes #3463
2017-11-07 18:13:23 -08:00
James Phillips 099b143838
Double-books the HTTP metrics w/ and w/o the "consul" prefix.
Fixes #3654
2017-11-07 16:32:45 -08:00
James Phillips 4a2cafe525
Adds HTTP/2 support to Consul's HTTPS server. (#3657)
* Refactors the HTTP listen path to create servers in the same spot.

* Adds HTTP/2 support to Consul's HTTPS server.

* Vendors Go HTTP/2 library and associated deps.
2017-11-07 15:06:59 -08:00
James Phillips aa199ab6ba
Makes the metrics ACL test call the right endpoint.
This also required setting up a proper in-mem sink so we don't get
metrics-related errors.

Fixes #3655
2017-11-06 21:50:04 -08:00
Preetha Appan b15d8db851 Sets tty in docker client back to true, as a potential fix for docker exec weirdness 2017-11-05 09:44:55 -06:00
Kyle Havlovitz d3dd2b1402
Move check definition to a sub-struct 2017-11-01 14:54:46 -07:00
Kyle Havlovitz dbab3cd5f6
Merge branch 'master' into esm-changes 2017-11-01 11:37:48 -07:00
Kyle Havlovitz c4375d5a47
Merge pull request #3622 from hashicorp/coordinate-node-endpoint
agent: add /v1/coordianate/node/:node endpoint
2017-11-01 11:35:50 -07:00
Kyle Havlovitz 021b2849c5
Remove redundant lines from coordinate test 2017-11-01 11:25:33 -07:00
Kyle Havlovitz b0536a96cc
Fill out the tests around coordinate/node functionality 2017-10-31 15:36:44 -07:00
Frank Schröder 874e350b2f config: add -config-format option (#3626)
* config: refactor ReadPath(s) methods without side-effects

Return the sources instead of modifying the state.

* config: clean data dir before every test

* config: add tests for config-file and config-dir

* config: add -config-format option

Starting with Consul 1.0 all config files must have a '.json' or '.hcl'
extension to make it unambigous how the data should be parsed. Some
automation tools generate temporary files by appending a random string
to the generated file which obfuscates the extension and prevents the
file type detection.

This patch adds a -config-format option which can be used to override
the auto-detection behavior by forcing all config files or all files
within a config directory independent of their extension to be
interpreted as of this format.

Fixes #3620
2017-10-31 17:30:01 -05:00
Frank Schröder a052255f86 vendor: update go-discover (#3634)
* vendor: update go-discover

Pull in providers:

 * Aliyun (Alibaba Cloud)
 * Digital Ocean
 * OpenStack (os)
 * Scaleway

* doc: use ... instead of xxx

* doc: strip trailing whitespace

* doc: add docs for aliyun, digitalocean, os and scaleway

* agent: fix test
2017-10-31 17:03:54 -05:00
Kyle Havlovitz 1e3b0d441b
Factor out registerNodes function 2017-10-31 13:34:49 -07:00
James Phillips 6bf55d16a2
Relaxes Autopilot promotion logic. (#3623)
* Relaxes Autopilot promotion logic.

When we defaulted the Raft protocol version to 3 in #3477 we made
the numPeers() routine more strict to only count voters (this is
more conservative and more correct). This had the side effect of
breaking rolling updates because it's at odds with the Autopilot
non-voter promotion logic.

That logic used to wait to only promote to maintain an odd quorum
of servers. During a rolling update (add one new server, wait, and
then kill an old server) the dead server cleanup would still count
the old server as a peer, which is conservative and the right thing
to do, and no longer count the non-voter. This would wait to promote,
so you could get into a stalemate. It is safer to promote early than
remove early, so by promoting as soon as possible we have chosen
that as the solution here.

Fixes #3611

* Gets rid of unnecessary extra not-a-voter check.
2017-10-31 15:16:56 -05:00
Frank Schroeder c72e6bdb37
docker: fix failing test 2017-10-31 09:26:34 +01:00
Frank Schroeder 7c0e5036a8
docker: render errors with %v since they can be nil 2017-10-31 09:19:20 +01:00
Kyle Havlovitz a1d14019dd
Add tests around coordinate update endpoint 2017-10-26 20:12:54 -07:00
Kyle Havlovitz 2392545adc
Merge branch 'coordinate-node-endpoint' of github.com:hashicorp/consul into esm-changes 2017-10-26 19:20:24 -07:00
Kyle Havlovitz 5589eadcf5
Added Coordinate.Node rpc endpoint and client api method 2017-10-26 19:16:40 -07:00
Frank Schroeder ca9aac746f
agent: add /v1/coordianate/node/:node endpoint
This patch adds a /v1/coordinate/node/:node endpoint to get the network
coordinates for a single node in the network.

Since Consul Enterprise supports network segments it is still possible
to receive mutiple entries for a single node - one per segment.
2017-10-26 14:24:42 +02:00
Frank Schroeder b4d8c26194
docker: add comment about "connection reset by peer" error 2017-10-26 12:14:19 +02:00
Frank Schroeder 164ec3ec39
docker: stop previous check on replace 2017-10-26 12:03:07 +02:00
Frank Schroeder e774b46f82
docker: close idle connections on stop 2017-10-26 12:02:39 +02:00
Frank Schroeder 94726ef105
docker: do not alloc a tty since this is not interactive 2017-10-26 11:56:54 +02:00
Frank Schroeder a1b47d99c1
docker: make sure to log the error when we fall through 2017-10-26 11:56:36 +02:00
Frank Schroeder 51a18c2557
docker: ignore "connection reset by peer"
The Docker agent closes the connection during read after we have
read the body. This causes a "connection reset by peer" even though
the command was successful.

We ignore that error here since we got the correct status code
and a response body.
2017-10-26 11:56:08 +02:00
Kyle Havlovitz ce4e8c46fa
Add deregister critical service field and refactor duration parsing 2017-10-25 19:17:41 -07:00
Kyle Havlovitz 291455f475
Added coordinate update http endpoint 2017-10-25 19:37:30 +02:00
Kyle Havlovitz d56936e27a
Added remaining HTTP health check fields to structs 2017-10-25 19:37:30 +02:00
Kyle Havlovitz a7c42a6c2a
Expose SkipNodeUpdate field and some health check info in the http api 2017-10-25 19:37:30 +02:00
Frank Schroeder d14adc6b4d
fix go vet issue 2017-10-25 19:30:35 +02:00
Frank Schroeder 96fdbd00a6
replace custom unique id with a UUID 2017-10-25 19:30:35 +02:00
Frank Schroeder 8f145559d8
Decouple the code that executes checks from the agent 2017-10-25 11:18:07 +02:00
Frank Schroeder 1cb8b0ffe3
local state: fix go vet issue 2017-10-23 10:56:05 +02:00
Frank Schroeder 7335c34c32
local state: remove stale comment 2017-10-23 10:56:05 +02:00
Frank Schroeder 3d547e30c7
local state: make test more robust 2017-10-23 10:56:05 +02:00
Frank Schroeder 52e73301f6
local state: clone check to avoid side effect 2017-10-23 10:56:05 +02:00
Frank Schroeder 6bc9d66192
local state: use synchronized access to internal maps 2017-10-23 10:56:05 +02:00
Frank Schroeder cc0499da3d
ae: do not trigger on Resume while holding the lock 2017-10-23 10:56:05 +02:00
Frank Schroeder 92f136de12
ae: add remaining test cases 2017-10-23 10:56:05 +02:00
Frank Schroeder 622ace2829
ae: refactor StateSyncer to state machine for better testing 2017-10-23 10:56:05 +02:00
Frank Schroeder 1212598ae2
ae: add test that we run a full before a partial sync 2017-10-23 10:56:05 +02:00
Frank Schroeder 4431e222fe
ae: make control flow more explicit 2017-10-23 10:56:05 +02:00
Frank Schroeder 3231385089
ae: fix typo in constructor name 2017-10-23 10:56:05 +02:00
Frank Schroeder 51daa96dfe
ae: add test for resume triggering SyncChanges 2017-10-23 10:56:05 +02:00
Frank Schroeder 92088d21e8
ae: add test for ifNotPausedRun 2017-10-23 10:56:05 +02:00
Frank Schroeder 11e172d1e9
ae: make stagger function pluggable for testing 2017-10-23 10:56:05 +02:00
Frank Schroeder e2452efed8
ae: restore previous pause/resume behavior 2017-10-23 10:56:04 +02:00
Frank Schroeder aba072bd1d
ae: ensure that syncs are blocked when paused 2017-10-23 10:56:04 +02:00
Frank Schroeder 58d52ac580
local state: rename Add{Check,Service}State to Set{Check,Service}State 2017-10-23 10:56:04 +02:00
Frank Schroeder e144f51b29
local state: move Metadata methods together 2017-10-23 10:56:04 +02:00
Frank Schroeder 4f9e05f634
local state: update documentation of updateSyncState 2017-10-23 10:56:04 +02:00
Frank Schroeder 41c7b0927e
local state: update comments 2017-10-23 10:56:04 +02:00
Frank Schroeder de57b16d99
local state: address review comments
* move non-blocking notification mechanism into ae.Trigger
* move Pause/Resume into separate type
2017-10-23 10:56:04 +02:00
Frank Schroeder 5c77c59501
local state: refactor TestAgentAntiEntropy_EnableTagOverride
Make intent clearer by being more explicit and adding some comments.
Use verify.Values to compare service entries.
2017-10-23 10:56:04 +02:00
Frank Schroeder 524981f367
local state: fix TestAgentAntiEntropy_EnableTagOverride
The test had a race condition where it relied on the first service to be
synced to the remote catalog which sometimes failed.
2017-10-23 10:56:04 +02:00
Frank Schroeder 1181aaee6a
local state: rename tests 2017-10-23 10:56:04 +02:00
Frank Schroeder 1602ac56b5
local state: drop retry loops from tests
Since the tests are now using synchronous calls for state syncing
we no longer need to use retry loops to wait for the changes to
propagate.
2017-10-23 10:56:04 +02:00
Frank Schroeder 7e3adc4549
agent: skip non-sensical TestCatalogRegister
It is not clear what this test is supposed to verify.
2017-10-23 10:56:04 +02:00
Frank Schroeder 71c74e62c7
local state: fix anti-entropy state tests
The anti-entropy tests relied on the side-effect of the StartSync()
method to perform a full sync instead of a partial sync. This lead to
multiple anti-entropy go routines being started unnecessary retry loops.

This change changes the behavior to perform synchronous full syncs when
necessary removing the need for all of the time.Sleep and most of the
retry loops.
2017-10-23 10:56:04 +02:00