Commit Graph

387 Commits

Author SHA1 Message Date
James Phillips 218ac4cb1e Obfuscates ACL tokens appearing in /v1/acl/<verb>/<token> APIs. (#3276)
* Obfuscates ACL tokens appearing in /v1/acl APIs.

* Makes test positively identify the desired strings.

* Adds an example and explanation of the regular expression.
2017-07-15 00:07:08 -07:00
James Phillips 872cf9ff95 Changes ACL clone response to 403 if not authorized, or if token doesn't exist. (#3275)
Fixes #1113
2017-07-14 20:43:30 -07:00
Kyle Havlovitz 78c3a86405
Add TLS setting to router areas 2017-07-14 17:38:08 -07:00
James Phillips 0881e46111 Cleans up version 8 ACLs in the agent and the docs. (#3248)
* Moves magic check and service constants into shared structs package.

* Removes the "consul" service from local state.

Since this service is added by the leader, it doesn't really make sense to
also keep it in local state (which requires special ACLs to configure), and
requires a bunch of special cases in the local state logic. This requires
fewer special cases and makes ACL bootstrapping cleaner.

* Makes coordinate update ACL log message a warning, similar to other AE warnings.

* Adds much more detailed examples for bootstrapping ACLs.

This can hopefully replace https://gist.github.com/slackpad/d89ce0e1cc0802c3c4f2d84932fa3234.
2017-07-13 22:33:47 -07:00
Frank Schroeder 764dabfcf7
agent: fix go vet issue 2017-07-11 07:13:46 -07:00
James Phillips 66edec5dfd Adds the ability to blacklist specific HTTP endpoints. (#3252) 2017-07-10 13:51:25 -07:00
James Phillips 7200b8cda8 UI cleanup follow up from #3245. (#3251)
* Removes unnecessary set for model component which will be null.

* Returns a 404 for a missing node, not a 200 with an empty response.

* Updates built-in web assets.
2017-07-10 09:40:00 -07:00
James Phillips aa11956d63 Changes the default ACL token type to "client" in web UI. (#3246)
* Changes the default ACL token type to "client".

* Updates built-in web assets.
2017-07-08 17:28:04 -07:00
James Phillips 86b1e64a33 Cleans up web UI and fixes ACL token "stuckness" issue. (#3245)
* Removes GitHub reference.

* Doesn't display ACL token on the unauthorized page.

* Removes useless fetch for nodes and cleans up comments.

* Provides a path to reset the ACL token when it's invalid.

This included making the settings page global so it's reachable, and adding
some more information about an error on the error page.

* Updates built-in web assets.
2017-07-08 17:16:05 -07:00
Frank Schroeder 1781fd311f address review comments 2017-07-07 09:22:34 +02:00
Frank Schroeder e4b40acc7e agent: remove unused code 2017-07-07 09:22:34 +02:00
Frank Schroeder 8c792ad57d agent: make TestClient_RPC_ConsulServerPing more robust 2017-07-07 09:22:34 +02:00
Frank Schroeder 4a4b91a2db agent: fix data races with registerEndpoint
Only register a different endpoint after it has been
fully created.
2017-07-07 09:22:34 +02:00
Frank Schroeder 19b937ba80 agent: make Reap test timing less aggressive 2017-07-07 09:22:34 +02:00
James Phillips a855d31f84 Adds a comment about flood joining. 2017-07-07 09:22:34 +02:00
James Phillips 5b5217528a Simplifies Serf dynamic port selection code.
This isn't racy, it's just a little dirty. The listen will happen and a port
will be selected and injected into the config once the Serf instance is
created, so we don't need the retry loop here.
2017-07-07 09:22:34 +02:00
James Phillips d8db4bc086 test: Changes WAN/LAN join confirmer to use port number vs. address.
This fixes TestServer_JoinSeparateLanAndWanAddresses which sets bogus
advertise addresses as part of the test. Port numbers uniquely identify
members since everything is running on localhost.
2017-07-07 09:22:34 +02:00
Frank Schroeder d92f70f313 test: make joinLAN/WAN reliable
only return if the members can see each other
2017-07-07 09:22:34 +02:00
Frank Schroeder 112bc19cd5 rpc: make TestServer_JoinSeparateLanAndWanAddresses more robust 2017-07-07 09:22:34 +02:00
Frank Schroeder ffd45f5da5 rpc: make TestClient_SnapshotRPC_TLS more robust 2017-07-07 09:22:34 +02:00
Frank Schroeder c218fdbc77 agent: make timing sensitive tests more robust
* make timing less aggressive
* mark timing tests as non-parallel
2017-07-07 09:22:34 +02:00
Frank Schroeder b12b914017 agent: fix TestCheckHTTP_TLSSkipVerify_true_pass
Make check timing less aggressive and give the test some time
to execute.
2017-07-07 09:22:34 +02:00
Frank Schroeder 7381a05d8d agent: do not modify agent config after NewAgent 2017-07-07 09:22:34 +02:00
Frank Schroeder cec9dcfee8 agent: fix pending data races between localState and agent
This patch creates a local config structure for the local state
which is independent from the agent but populated from its
configuration. This avoids data races between the agent configuration
which can change during tests and concurrent go routines using the
configuraiton at the same time.
2017-07-07 09:22:34 +02:00
Frank Schroeder 00f6ba298e dns: fix data race in TestDNS_ServiceLookup_FilterACL
The agent config cannot be modified after start.
2017-07-07 09:22:34 +02:00
Frank Schroeder bcd2301b81 agent: fix data race in TestAgentAntiEntropy_EnableTagOverride 2017-07-07 09:22:34 +02:00
Frank Schroeder 7f7c0ad65e agent: clone partial consul config
The agent configuration for the consul server is a partial configuration
which needs to be cloned to avoid data races.

This is a stop-gap measure before moving the configuration into
a separate package.
2017-07-07 09:22:34 +02:00
Frank Schroeder 727b6444ad dns: fix data races in DNS compression tests
Make the DisableCompression value configurable at runtime
to allow tests to change it without restarting/recreating
the server.
2017-07-07 09:22:34 +02:00
Frank Schroeder 0763788b82 agent: fix data race between consul server and local state 2017-07-07 09:22:34 +02:00
Frank Schroeder 119f6a1ed7 rpc: monkey patch fix for data races for localState
The tests that use the localState of the agent access the internal
variables and call methods which are not guarded by locks creating
data races in tests. While the use of internal variables is somewhat
easy to spot the fact that not all methods are thread-safe is a
surprise.

A proper fix requires the localState struct to be moved into its own
package so that tests in the agent can only access the external
interface.

However, the localState is currently dependent on the agent.Config
which would create a circular dependency. Therefore, the Config
struct needs to be moved first for this to happen.

This patch literally monkey patches the use of the lock around the
cases which have data races and marks them with a
// todo(fs): data race comment.
2017-07-07 09:22:34 +02:00
Frank Schroeder 2159d499e3 rpc: try shutting down leader first to avoid hang in TestLeader_LeftServer 2017-07-07 09:22:34 +02:00
Frank Schroeder f12fac278e rpc: fix logging and try quicker timing of TestServer_JoinSeparateLanAndWanAddresses 2017-07-07 09:22:34 +02:00
Frank Schroeder bae4b1d045 rpc: less agressive raft timeouts
Allowing more time for raft to consolidate should
drop the number of leader elections.
2017-07-07 09:22:34 +02:00
Frank Schroeder 457b98a099 rpc: run agent/consul tests in parallel 2017-07-07 09:22:34 +02:00
Frank Schroeder 13eeeb720d rpc: refactor sessionTimers and fix racy tests
The sessionTimers map was secured by a lock which wasn't used
properly in the tests. This lead to data races and failing tests
when accessing the length or the members of the map.

This patch adds a separate SessionTimers struct which is safe
for concurrent use and which ecapsulates the behavior of the
sessionTimers map.
2017-07-07 09:22:34 +02:00
Frank Schroeder 05f756853e rpc: fix TestServer_Leave
wait for the leader election.
2017-07-07 09:22:34 +02:00
Frank Schroeder 583959392b rpc: fix TestSession_Renew
make the timing less tight
2017-07-07 09:22:34 +02:00
Frank Schroeder ff2c29c0be rpc: fix TestReadyForConsistentRead
timing was too tight. Standardized name.
2017-07-07 09:22:34 +02:00
Frank Schroeder fcab525053 rpc: fix for 'no leader' in TLS tests
Ensure both servers know about each other before looking
for a leader.
2017-07-07 09:22:34 +02:00
Frank Schroeder b2a71fd8b0 rpc: fix TestServer_JoinWAN_Flood
The second server in the first data center should not be
in bootstrap mode.
2017-07-07 09:22:34 +02:00
Frank Schroeder 8369b6cb9d rpc: provide unique node names for server and client 2017-07-07 09:22:34 +02:00
Frank Schroeder 534977239b rpc: prefix log output with test name 2017-07-07 09:22:34 +02:00
Frank Schroeder c8ef588d8d rpc: discover serf wan port before starting serf lan
When using dynamic ports for the serf clusters then
the actual bind port of the serf WAN cluster needs to
be discovered before the serf LAN cluster is started
since the serf LAN cluster announces the port of the WAN
cluster.
2017-07-07 09:22:34 +02:00
Frank Schroeder 53eab7e970 rpc: bind rpc test server to port 0 2017-07-07 09:22:34 +02:00
Frank Schroeder e9e2c599db rpc: refactor: unify test server setup 2017-07-07 09:22:34 +02:00
Frank Schroeder c803146550 rpc: fix typos 2017-07-07 09:22:34 +02:00
Frank Schroeder a0368e3827 agent: refactor: log to stderr during tests 2017-07-07 09:22:34 +02:00
Frank Schroeder 2afafe6911 agent: refactor: use handler for test http tls server 2017-07-07 09:22:34 +02:00
Frank Schroeder 63447a0cf3 agent: refactor: make address translation part of the agent 2017-07-07 09:22:34 +02:00
Preetha Appan f549c06764 Rename to raftNotifyCh, fix typo 2017-07-06 09:10:36 -05:00
Preetha Appan f2171a6720 Fixes deadlock between barrier write and leader notify channel read . Fixes #3230 2017-07-05 17:09:18 -05:00
Grégoire Seux 940b1513a6 Correctly forward Host header in healthcheck (#3203)
Host header must be set explicitely on http requests

Change-Id: I91a32f0fb1ec3fbc713adf0e10869797e91172c7
Signed-off-by: Grégoire Seux <g.seux@criteo.com>
2017-06-29 16:26:08 -07:00
Preetha Appan 07db760d53 Fix missing formatting directive causing go vet to fail 2017-06-27 16:32:38 -05:00
Frank Schroeder 6dd68f7ad7
Revert "agent: add allowStale option for HTTP API (#3142)"
This reverts commit 1e0fd27a74.
2017-06-27 07:04:55 +02:00
Frank Schröder 4b51d00458 agent: fix DNS recursor tests (#3190)
The makeRecursor function was using an unreliable mechanism
to start a server with a random port. This patch changes this
so that the server starts on port 0 to let the kernel pick
a free port.

In addition, to similar functions for starting a test DNS
server were folded into one.
2017-06-25 10:42:37 -07:00
James Phillips 4a3604a3ee
Removes some useless comments. 2017-06-25 10:32:35 -07:00
James Phillips 6977e40077 Fixes watch tracking during reloads and fixes address issue. (#3189)
This patch fixes watch registration through the config file and a broken log line when the watch registration fails. It also plumbs all the watch loading through a common function and tweaks the
unit test to create the watch before the reload.
2017-06-24 12:52:41 -07:00
James Phillips 380c8b957d Changes host-based node IDs from opt-out to opt-in. (#3187) 2017-06-24 09:36:53 -07:00
James Phillips 42f60b04bb Revert "discover: move instance discover code into separate package (#3144)" (#3180)
This reverts commit 26bfb2d00a.
2017-06-23 01:38:55 -07:00
James Phillips e4b11682bc Fixes broken HTTP header and method for health checks. (#3178)
* Fixes broken HTTP header and method for health checks.
* Adds a fuzz utility and test to make sure copy is complete.
2017-06-23 01:15:48 -07:00
wojtkiewicz 1e0fd27a74
agent: add allowStale option for HTTP API (#3142)
This patch adds an "allowStale" option to the HTTP API
configuration which allows stale reads to provide linear
read scalability.

Fixes #3142
2017-06-22 10:31:13 +02:00
wojtkiewicz 37785028be
agent: add "http_config"
This patch adds an "http_config" object to the config file
and moves the "http_api_response_headers" option there.

"http_api_response_headers" is now deprecated in favor of
"http_config.response_headers"
2017-06-22 10:31:11 +02:00
James Phillips 0e695aef94 Fixes checked in web assets and associated build scripts. (#3173) 2017-06-21 14:43:07 -07:00
Frank Schröder 26bfb2d00a discover: move instance discover code into separate package (#3144)
This patch moves the code that discovers instances from metadata
information to github.com/hashicorp/go-discover with
sub-packages for each provider.
2017-06-21 10:40:38 +02:00
Frank Schröder 31a310f551 agent: notify systemd after JoinLAN (#2121)
This patch adds support for notifying systemd via the
NOTIFY_SOCKET by sending 'READY=1' to the socket after
a successful JoinLAN.

Fixes #2121
2017-06-21 06:43:55 +02:00
Frank Schroeder ea5b0f2c7c agent: fix 'consul leave' shutdown race (#2880)
When the agent is triggered to shutdown via an external 'consul leave'
command delivered via the HTTP API then the client expects to receive a
response when the agent is down. This creates a race on when to shutdown
the agent itself like the RPC server, the checks and the state and the
external endpoints like DNS and HTTP.

This patch splits the shutdown process into two parts:

 * shutdown the agent
 * shutdown the endpoints (http and dns)

They can be executed multiple times, concurrently and in any order but
should be executed first agent, then endpoints to provide consistent
behavior across all use cases. Both calls have to be executed for a
proper shutdown.

This could be partially hidden in a single function but would introduce
some magic that happens behind the scenes which one has to know of but
isn't obvious.

Fixes #2880
2017-06-21 05:52:51 +02:00
Frank Schroeder 7abe308c66 agent: drop unused constant 2017-06-21 05:42:39 +02:00
Frank Schroeder c4fc581e07 agent: make registerEndpoint private
This is only used for testing.
2017-06-21 05:42:39 +02:00
Frank Schroeder 2b41f2e3a3 agent: make the RPC endpoint overwrite mechanism more transparent
This patch hides the RPC handler overwrite mechanism from the
rest of the code so that it works in all cases and that there
is no cooperation required from the tested code, i.e. we can
drop a.getEndpoint().
2017-06-21 05:42:39 +02:00
Frank Schroeder e15f9f9d90 agent: rename agent var 2017-06-21 05:42:39 +02:00
Frank Schroeder c49a15d0f3 agent: move structs into consul/structs pkg
* CheckDefinition
 * ServiceDefinition
 * CheckType
2017-06-21 05:42:39 +02:00
Frank Schroeder 4273fb8444 agent: move NotifyGroup into the agent pkg 2017-06-21 05:42:39 +02:00
Frank Schroeder 82a132da60 agent: move conn pool for muxed connections into separate pkg 2017-06-21 05:42:39 +02:00
Frank Schroeder 80971c8a85 agent: move the SnapshotReplyFn out of the way
When splitting up the consul package into server and client
the SnapshotReplyFn needs to be in a separate package to avoid
a circular dependency.
2017-06-21 05:42:39 +02:00
Frank Schroeder 04b9392b00 agent: use the delegate interface for local state 2017-06-21 05:42:39 +02:00
Frank Schroeder d77d2be13e agent: rename clientServer interface to delegate 2017-06-21 05:42:39 +02:00
preetapan 0c2a83e1ec Merge pull request #3154 from hashicorp/issue_2644_redux
Fix stale reads on server startup. Consistent reads will now wait for up to config.RPCHoldTimeout for the server to get past its raft log, before returning an error. Servers that are starting up will eventually catch up. 
This fixes issue #2644
2017-06-20 19:47:12 -05:00
Preetha Appan f658231ab9 Minor fixes per code review 2017-06-20 19:43:07 -05:00
Frank Schroeder b083ce17c7
Revert "agent: fix 'consul leave' shutdown race (#2880)"
This reverts commit 90c83a32b5.
2017-06-19 21:34:08 +02:00
Frank Schroeder 90c83a32b5 agent: fix 'consul leave' shutdown race (#2880)
When the agent is triggered to shutdown via an external 'consul leave'
command delivered via the HTTP API then the client expects to receive a
response when the agent is down. This creates a race on when to shutdown
the agent itself like the RPC server, the checks and the state and the
external endpoints like DNS and HTTP. Ideally, the external endpoints
should be shutdown before the internal state but if the goal is to
respond reliably that the agent is down then this is not possible.

This patch splits the agent shutdown into two parts implemented in a
single method to keep it simple and unambiguos for the caller. The first
stage shuts down the internal state, checks, RPC server, ...
synchronously and then triggers the shutdown of the external endpoints
asychronously. This way the caller is guaranteed that the internal state
services are down when Shutdown returns and there remains enough time to
send a response.

Fixes #2880
2017-06-19 21:24:26 +02:00
Preetha Appan b3b2e9dcb4 Added unit test to verify consistentRead method behavior 2017-06-16 11:58:12 -05:00
Preetha Appan 44f5086873 Code review feedback, fixed major logic bug 2017-06-16 10:49:54 -05:00
Preetha Appan 72af7b9bc4 Redo bug fix for stale reads on server startup, leveraging RPCHOldtimeout instead of maxQueryTime, plus tests 2017-06-15 22:41:30 -05:00
Kyle Havlovitz 5d99ee80ca Add an option to disable keyring file (#3145)
Also disables keyring file in dev mode.
2017-06-15 15:24:04 -07:00
Seth Vargo a8f1e7ebf9 Add EDNS0 support (#3131)
This is a refactor of GH-1980. Originally I tried to do a straight
rebase, but the code has changed too much.
2017-06-14 16:22:54 -07:00
Preetha Appan cf5006c4f4 Fixed static asset build target and checked in new executable for assetfs 2017-06-12 12:57:02 -05:00
Frank Schroeder 1c75cf1af5 pkg refactor
command/agent/*                  -> agent/*
    command/consul/*                 -> agent/consul/*
    command/agent/command{,_test}.go -> command/agent{,_test}.go
    command/base/command.go          -> command/base.go
    command/base/*                   -> command/*
    commands.go                      -> command/commands.go

The script which did the refactor is:

(
	cd $GOPATH/src/github.com/hashicorp/consul
	git mv command/agent/command.go command/agent.go
	git mv command/agent/command_test.go command/agent_test.go
	git mv command/agent/flag_slice_value{,_test}.go command/
	git mv command/agent .
	git mv command/base/command.go command/base.go
	git mv command/base/config_util{,_test}.go command/
	git mv commands.go command/
	git mv consul agent
	rmdir command/base/

	gsed -i -e 's|package agent|package command|' command/agent{,_test}.go
	gsed -i -e 's|package agent|package command|' command/flag_slice_value{,_test}.go
	gsed -i -e 's|package base|package command|' command/base.go command/config_util{,_test}.go
	gsed -i -e 's|package main|package command|' command/commands.go

	gsed -i -e 's|base.Command|BaseCommand|' command/commands.go
	gsed -i -e 's|agent.Command|AgentCommand|' command/commands.go
	gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/commands.go
	gsed -i -e 's|base\.||' command/commands.go
	gsed -i -e 's|command\.||' command/commands.go

	gsed -i -e 's|command|c|' main.go
	gsed -i -e 's|range Commands|range command.Commands|' main.go
	gsed -i -e 's|Commands: Commands|Commands: command.Commands|' main.go

	gsed -i -e 's|base\.BoolValue|BoolValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.DurationValue|DurationValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.StringValue|StringValue|' command/operator_autopilot_set.go
	gsed -i -e 's|base\.UintValue|UintValue|' command/operator_autopilot_set.go

	gsed -i -e 's|\bCommand\b|BaseCommand|' command/base.go
	gsed -i -e 's|BaseCommand Options|Command Options|' command/base.go
	gsed -i -e 's|base.Command|BaseCommand|' command/*.go
	gsed -i -e 's|c\.Command|c.BaseCommand|g' command/*.go
	gsed -i -e 's|\tCommand:|\tBaseCommand:|' command/*_test.go
	gsed -i -e 's|base\.||' command/*_test.go

	gsed -i -e 's|\bCommand\b|AgentCommand|' command/agent{,_test}.go
	gsed -i -e 's|cmd.AgentCommand|cmd.BaseCommand|' command/agent.go

	gsed -i -e 's|cli.AgentCommand = new(Command)|cli.Command = new(AgentCommand)|' command/agent_test.go
	gsed -i -e 's|exec.AgentCommand|exec.Command|' command/agent_test.go
	gsed -i -e 's|exec.BaseCommand|exec.Command|' command/agent_test.go
	gsed -i -e 's|NewTestAgent|agent.NewTestAgent|' command/agent_test.go
	gsed -i -e 's|= TestConfig|= agent.TestConfig|' command/agent_test.go
	gsed -i -e 's|: RetryJoin|: agent.RetryJoin|' command/agent_test.go

	gsed -i -e 's|\.\./\.\./|../|' command/config_util_test.go

	gsed -i -e 's|\bverifyUniqueListeners|VerifyUniqueListeners|' agent/config{,_test}.go command/agent.go
	gsed -i -e 's|\bserfLANKeyring\b|SerfLANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
	gsed -i -e 's|\bserfWANKeyring\b|SerfWANKeyring|g' agent/{agent,keyring,testagent}.go command/agent.go
	gsed -i -e 's|\bNewAgent\b|agent.New|g' command/agent{,_test}.go
	gsed -i -e 's|\bNewAgent|New|' agent/{acl_test,agent,testagent}.go

	gsed -i -e 's|\bAgent\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bBool\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bDefaultConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bDevConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bMergeConfig\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bReadConfigPaths\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bParseMetaPair\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bSerfLANKeyring\b|agent.&|g' command/agent{,_test}.go
	gsed -i -e 's|\bSerfWANKeyring\b|agent.&|g' command/agent{,_test}.go

	gsed -i -e 's|circonus\.agent|circonus|g' command/agent{,_test}.go
	gsed -i -e 's|logger\.agent|logger|g' command/agent{,_test}.go
	gsed -i -e 's|metrics\.agent|metrics|g' command/agent{,_test}.go
	gsed -i -e 's|// agent.Agent|// agent|' command/agent{,_test}.go
	gsed -i -e 's|a\.agent\.Config|a.Config|' command/agent{,_test}.go

	gsed -i -e 's|agent\.AppendSliceValue|AppendSliceValue|' command/{configtest,validate}.go

	gsed -i -e 's|consul/consul|agent/consul|' GNUmakefile

	gsed -i -e 's|\.\./test|../../test|' agent/consul/server_test.go

	# fix imports
	f=$(grep -rl 'github.com/hashicorp/consul/command/agent' * | grep '\.go')
	gsed -i -e 's|github.com/hashicorp/consul/command/agent|github.com/hashicorp/consul/agent|' $f
	goimports -w $f

	f=$(grep -rl 'github.com/hashicorp/consul/consul' * | grep '\.go')
	gsed -i -e 's|github.com/hashicorp/consul/consul|github.com/hashicorp/consul/agent/consul|' $f
	goimports -w $f

	goimports -w command/*.go main.go
)
2017-06-10 18:52:45 +02:00