status-im
/
consul
mirror of https://github.com/status-im/consul.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge branch 'master' into f-gce-discovery

This commit is contained in:
Kyle Havlovitz 2017-01-11 22:57:07 -05:00
parent dbc72eaac0 023b6533e0
commit 5ddea8a5df
No known key found for this signature in database
GPG Key ID: 8A5E6B173056AD6C
95 changed files with 9011 additions and 1194 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
CHANGELOG.md
View File

@ -1,4 +1,21 @@
## 0.7.2 (UNRELEASED)
## 0.7.3 (UNRELEASED)
FEATURES:
* **KV Import/Export CLI:** `consul kv export` and `consul kv import` can be used to move parts of the KV tree between disconnected consul clusters, using JSON as the intermediate representation. [GH-2633]
* **Node Metadata:** Support for assigning user-defined metadata key/value pairs to nodes has been added. This can be viewed when looking up node info, and can be used to filter the results of the `/v1/catalog/nodes` or `/v1/catalog/services` endpoints. For more information, see the [Node Meta](https://www.consul.io/docs/agent/options.html#_node_meta) section of the agent configuration. [GH-154]
IMPROVEMENTS:
* cli: `consul kv get` now has a `-base64` flag to base 64 encode the value. [GH-2631]
* cli: `consul kv put` now has a `-base64` flag for setting values which are base 64 encoded. [GH-2632]
* ui: Added a notice that JS is required when viewing the web UI with JS disabled. [GH-2636]
BUG FIXES:
* cli: Fixed a panic when an empty quoted argument was given to `consul kv put`. [GH-2635]
## 0.7.2 (December 19, 2016)
FEATURES:
@ -7,14 +24,22 @@ FEATURES:
* **Reload API:** A new `/v1/agent/reload` HTTP endpoint was added for triggering a reload of the agent's configuration. See the [Reload Endpoint](https://www.consul.io/docs/agent/http/agent.html#agent_reload) for more details. [GH-2516]
* **Leave API:** A new `/v1/agent/leave` HTTP endpoint was added for causing an agent to gracefully shutdown and leave the cluster (previously, only `force-leave` was present in the HTTP API). See the [Leave Endpoint](https://www.consul.io/docs/agent/http/agent.html#agent_leave) for more details. [GH-2516]
* **Bind Address Templates (beta):** Consul agents now allow [go-sockaddr/template](https://godoc.org/github.com/hashicorp/go-sockaddr/template) syntax to be used for any bind address configuration (`advertise_addr`, `bind_addr`, `client_addr`, and others). This allows for easy creation of immutable images for Consul that can fetch their own address based on an interface name, network CIDR, address family from an actual RFC number, and many other possible schemes. This feature is in beta and we may tweak the template syntax before final release, but we encourage the community to try this and provide feedback. [GH-2563]
* **Complete ACL Coverage (beta):** Consul 0.8 will feature complete ACL coverage for all of Consul. To ease the transition to the new policies, a beta version of complete ACL support was added to help with testing and migration to the new features. Please see the [ACLs Internals Guide](https://www.consul.io/docs/internals/acl.html#version_8_acls) for more details. [GH-2594, GH-2592, GH-2590]
IMPROVEMENTS:
* agent: Defaults to `?pretty` JSON for HTTP API requests when in `-dev` mode. [GH-2518]
* agent: Updated Circonus metrics library and added new Circonus configration options for Consul for customizing check display name and tags. [GH-2555]
* agent: Added a checksum to UDP gossip messages to guard against packet corruption. [GH-2574]
* agent: Check whether a snapshot needs to be taken more often (every 5 seconds instead of 2 minutes) to keep the raft file smaller and to avoid doing huge truncations when writing lots of entries very quickly. [GH-2591]
* agent: Allow gossiping to suspected/recently dead nodes. [GH-2593]
* agent: Changed the gossip suspicion timeout to grow smoothly as the number of nodes grows. [GH-2593]
* agent: Added a deprecation notice for Atlas features to the CLI and docs. [GH-2597]
* agent: Give a better error message when the given data-dir is not a directory. [GH-2529]
BUG FIXES:
* agent: Fixed a panic when SIGPIPE signal was received. [GH-2404]
* api: Added missing Raft index fields to `CatalogService` structure. [GH-2366]
* api: Added missing notes field to `AgentServiceCheck` structure. [GH-2336]
* api: Changed type of `AgentServiceCheck.TLSSkipVerify` from `string` to `bool`. [GH-2530]

444
acl/acl.go
View File

@ -35,6 +35,26 @@ func init() {
// ACL is the interface for policy enforcement.
type ACL interface {
// ACLList checks for permission to list all the ACLs
ACLList() bool
// ACLModify checks for permission to manipulate ACLs
ACLModify() bool
// AgentRead checks for permission to read from agent endpoints for a
// given node.
AgentRead(string) bool
// AgentWrite checks for permission to make changes via agent endpoints
// for a given node.
AgentWrite(string) bool
// EventRead determines if a specific event can be queried.
EventRead(string) bool
// EventWrite determines if a specific event may be fired.
EventWrite(string) bool
// KeyRead checks for permission to read a given key
KeyRead(string) bool
@ -46,26 +66,6 @@ type ACL interface {
// that deny a write.
KeyWritePrefix(string) bool
// ServiceWrite checks for permission to read a given service
ServiceWrite(string) bool
// ServiceRead checks for permission to read a given service
ServiceRead(string) bool
// EventRead determines if a specific event can be queried.
EventRead(string) bool
// EventWrite determines if a specific event may be fired.
EventWrite(string) bool
// PrepardQueryRead determines if a specific prepared query can be read
// to show its contents (this is not used for execution).
PreparedQueryRead(string) bool
// PreparedQueryWrite determines if a specific prepared query can be
// created, modified, or deleted.
PreparedQueryWrite(string) bool
// KeyringRead determines if the encryption keyring used in
// the gossip layer can be read.
KeyringRead() bool
@ -73,6 +73,13 @@ type ACL interface {
// KeyringWrite determines if the keyring can be manipulated
KeyringWrite() bool
// NodeRead checks for permission to read (discover) a given node.
NodeRead(string) bool
// NodeWrite checks for permission to create or update (register) a
// given node.
NodeWrite(string) bool
// OperatorRead determines if the read-only Consul operator functions
// can be used.
OperatorRead() bool
@ -81,11 +88,27 @@ type ACL interface {
// functions can be used.
OperatorWrite() bool
// ACLList checks for permission to list all the ACLs
ACLList() bool
// PrepardQueryRead determines if a specific prepared query can be read
// to show its contents (this is not used for execution).
PreparedQueryRead(string) bool
// ACLModify checks for permission to manipulate ACLs
ACLModify() bool
// PreparedQueryWrite determines if a specific prepared query can be
// created, modified, or deleted.
PreparedQueryWrite(string) bool
// ServiceRead checks for permission to read a given service
ServiceRead(string) bool
// ServiceWrite checks for permission to create or update a given
// service
ServiceWrite(string) bool
// SessionRead checks for permission to read sessions for a given node.
SessionRead(string) bool
// SessionWrite checks for permission to create sessions for a given
// node.
SessionWrite(string) bool
// Snapshot checks for permission to take and restore snapshots.
Snapshot() bool
@ -99,6 +122,30 @@ type StaticACL struct {
defaultAllow bool
}
func (s *StaticACL) ACLList() bool {
return s.allowManage
}
func (s *StaticACL) ACLModify() bool {
return s.allowManage
}
func (s *StaticACL) AgentRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) AgentWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) EventRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) EventWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) KeyRead(string) bool {
return s.defaultAllow
}
@ -111,30 +158,6 @@ func (s *StaticACL) KeyWritePrefix(string) bool {
return s.defaultAllow
}
func (s *StaticACL) ServiceRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) ServiceWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) EventRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) EventWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) PreparedQueryRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) PreparedQueryWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) KeyringRead() bool {
return s.defaultAllow
}
@ -143,6 +166,14 @@ func (s *StaticACL) KeyringWrite() bool {
return s.defaultAllow
}
func (s *StaticACL) NodeRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) NodeWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) OperatorRead() bool {
return s.defaultAllow
}
@ -151,12 +182,28 @@ func (s *StaticACL) OperatorWrite() bool {
return s.defaultAllow
}
func (s *StaticACL) ACLList() bool {
return s.allowManage
func (s *StaticACL) PreparedQueryRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) ACLModify() bool {
return s.allowManage
func (s *StaticACL) PreparedQueryWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) ServiceRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) ServiceWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) SessionRead(string) bool {
return s.defaultAllow
}
func (s *StaticACL) SessionWrite(string) bool {
return s.defaultAllow
}
func (s *StaticACL) Snapshot() bool {
@ -199,12 +246,21 @@ type PolicyACL struct {
// no matching rule.
parent ACL
// agentRules contains the agent policies
agentRules *radix.Tree
// keyRules contains the key policies
keyRules *radix.Tree
// nodeRules contains the node policies
nodeRules *radix.Tree
// serviceRules contains the service policies
serviceRules *radix.Tree
// sessionRules contains the session policies
sessionRules *radix.Tree
// eventRules contains the user event policies
eventRules *radix.Tree
@ -225,22 +281,40 @@ type PolicyACL struct {
func New(parent ACL, policy *Policy) (*PolicyACL, error) {
p := &PolicyACL{
parent: parent,
agentRules: radix.New(),
keyRules: radix.New(),
nodeRules: radix.New(),
serviceRules: radix.New(),
sessionRules: radix.New(),
eventRules: radix.New(),
preparedQueryRules: radix.New(),
}
// Load the agent policy
for _, ap := range policy.Agents {
p.agentRules.Insert(ap.Node, ap.Policy)
}
// Load the key policy
for _, kp := range policy.Keys {
p.keyRules.Insert(kp.Prefix, kp.Policy)
}
// Load the node policy
for _, np := range policy.Nodes {
p.nodeRules.Insert(np.Name, np.Policy)
}
// Load the service policy
for _, sp := range policy.Services {
p.serviceRules.Insert(sp.Name, sp.Policy)
}
// Load the session policy
for _, sp := range policy.Sessions {
p.sessionRules.Insert(sp.Node, sp.Policy)
}
// Load the event policy
for _, ep := range policy.Events {
p.eventRules.Insert(ep.Event, ep.Policy)
@ -260,6 +334,88 @@ func New(parent ACL, policy *Policy) (*PolicyACL, error) {
return p, nil
}
// ACLList checks if listing of ACLs is allowed
func (p *PolicyACL) ACLList() bool {
return p.parent.ACLList()
}
// ACLModify checks if modification of ACLs is allowed
func (p *PolicyACL) ACLModify() bool {
return p.parent.ACLModify()
}
// AgentRead checks for permission to read from agent endpoints for a given
// node.
func (p *PolicyACL) AgentRead(node string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.agentRules.LongestPrefix(node)
if ok {
switch rule {
case PolicyRead, PolicyWrite:
return true
default:
return false
}
}
// No matching rule, use the parent.
return p.parent.AgentRead(node)
}
// AgentWrite checks for permission to make changes via agent endpoints for a
// given node.
func (p *PolicyACL) AgentWrite(node string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.agentRules.LongestPrefix(node)
if ok {
switch rule {
case PolicyWrite:
return true
default:
return false
}
}
// No matching rule, use the parent.
return p.parent.AgentWrite(node)
}
// Snapshot checks if taking and restoring snapshots is allowed.
func (p *PolicyACL) Snapshot() bool {
return p.parent.Snapshot()
}
// EventRead is used to determine if the policy allows for a
// specific user event to be read.
func (p *PolicyACL) EventRead(name string) bool {
// Longest-prefix match on event names
if _, rule, ok := p.eventRules.LongestPrefix(name); ok {
switch rule {
case PolicyRead, PolicyWrite:
return true
default:
return false
}
}
// Nothing matched, use parent
return p.parent.EventRead(name)
}
// EventWrite is used to determine if new events can be created
// (fired) by the policy.
func (p *PolicyACL) EventWrite(name string) bool {
// Longest-prefix match event names
if _, rule, ok := p.eventRules.LongestPrefix(name); ok {
return rule == PolicyWrite
}
// No match, use parent
return p.parent.EventWrite(name)
}
// KeyRead returns if a key is allowed to be read
func (p *PolicyACL) KeyRead(key string) bool {
// Look for a matching rule
@ -327,10 +483,43 @@ func (p *PolicyACL) KeyWritePrefix(prefix string) bool {
return p.parent.KeyWritePrefix(prefix)
}
// ServiceRead checks if reading (discovery) of a service is allowed
func (p *PolicyACL) ServiceRead(name string) bool {
// KeyringRead is used to determine if the keyring can be
// read by the current ACL token.
func (p *PolicyACL) KeyringRead() bool {
switch p.keyringRule {
case PolicyRead, PolicyWrite:
return true
case PolicyDeny:
return false
default:
return p.parent.KeyringRead()
}
}
// KeyringWrite determines if the keyring can be manipulated.
func (p *PolicyACL) KeyringWrite() bool {
if p.keyringRule == PolicyWrite {
return true
}
return p.parent.KeyringWrite()
}
// OperatorRead determines if the read-only operator functions are allowed.
func (p *PolicyACL) OperatorRead() bool {
switch p.operatorRule {
case PolicyRead, PolicyWrite:
return true
case PolicyDeny:
return false
default:
return p.parent.OperatorRead()
}
}
// NodeRead checks if reading (discovery) of a node is allowed
func (p *PolicyACL) NodeRead(name string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.serviceRules.LongestPrefix(name)
_, rule, ok := p.nodeRules.LongestPrefix(name)
if ok {
switch rule {
@ -342,13 +531,13 @@ func (p *PolicyACL) ServiceRead(name string) bool {
}
// No matching rule, use the parent.
return p.parent.ServiceRead(name)
return p.parent.NodeRead(name)
}
// ServiceWrite checks if writing (registering) a service is allowed
func (p *PolicyACL) ServiceWrite(name string) bool {
// NodeWrite checks if writing (registering) a node is allowed
func (p *PolicyACL) NodeWrite(name string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.serviceRules.LongestPrefix(name)
_, rule, ok := p.nodeRules.LongestPrefix(name)
if ok {
switch rule {
@ -360,36 +549,16 @@ func (p *PolicyACL) ServiceWrite(name string) bool {
}
// No matching rule, use the parent.
return p.parent.ServiceWrite(name)
return p.parent.NodeWrite(name)
}
// EventRead is used to determine if the policy allows for a
// specific user event to be read.
func (p *PolicyACL) EventRead(name string) bool {
// Longest-prefix match on event names
if _, rule, ok := p.eventRules.LongestPrefix(name); ok {
switch rule {
case PolicyRead, PolicyWrite:
return true
default:
return false
}
// OperatorWrite determines if the state-changing operator functions are
// allowed.
func (p *PolicyACL) OperatorWrite() bool {
if p.operatorRule == PolicyWrite {
return true
}
// Nothing matched, use parent
return p.parent.EventRead(name)
}
// EventWrite is used to determine if new events can be created
// (fired) by the policy.
func (p *PolicyACL) EventWrite(name string) bool {
// Longest-prefix match event names
if _, rule, ok := p.eventRules.LongestPrefix(name); ok {
return rule == PolicyWrite
}
// No match, use parent
return p.parent.EventWrite(name)
return p.parent.OperatorWrite()
}
// PreparedQueryRead checks if reading (listing) of a prepared query is
@ -430,59 +599,74 @@ func (p *PolicyACL) PreparedQueryWrite(prefix string) bool {
return p.parent.PreparedQueryWrite(prefix)
}
// KeyringRead is used to determine if the keyring can be
// read by the current ACL token.
func (p *PolicyACL) KeyringRead() bool {
switch p.keyringRule {
case PolicyRead, PolicyWrite:
return true
case PolicyDeny:
return false
default:
return p.parent.KeyringRead()
// ServiceRead checks if reading (discovery) of a service is allowed
func (p *PolicyACL) ServiceRead(name string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.serviceRules.LongestPrefix(name)
if ok {
switch rule {
case PolicyRead, PolicyWrite:
return true
default:
return false
}
}
// No matching rule, use the parent.
return p.parent.ServiceRead(name)
}
// KeyringWrite determines if the keyring can be manipulated.
func (p *PolicyACL) KeyringWrite() bool {
if p.keyringRule == PolicyWrite {
return true
// ServiceWrite checks if writing (registering) a service is allowed
func (p *PolicyACL) ServiceWrite(name string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.serviceRules.LongestPrefix(name)
if ok {
switch rule {
case PolicyWrite:
return true
default:
return false
}
}
return p.parent.KeyringWrite()
// No matching rule, use the parent.
return p.parent.ServiceWrite(name)
}
// OperatorRead determines if the read-only operator functions are allowed.
func (p *PolicyACL) OperatorRead() bool {
switch p.operatorRule {
case PolicyRead, PolicyWrite:
return true
case PolicyDeny:
return false
default:
return p.parent.OperatorRead()
// SessionRead checks for permission to read sessions for a given node.
func (p *PolicyACL) SessionRead(node string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.sessionRules.LongestPrefix(node)
if ok {
switch rule {
case PolicyRead, PolicyWrite:
return true
default:
return false
}
}
// No matching rule, use the parent.
return p.parent.SessionRead(node)
}
// OperatorWrite determines if the state-changing operator functions are
// allowed.
func (p *PolicyACL) OperatorWrite() bool {
if p.operatorRule == PolicyWrite {
return true
// SessionWrite checks for permission to create sessions for a given node.
func (p *PolicyACL) SessionWrite(node string) bool {
// Check for an exact rule or catch-all
_, rule, ok := p.sessionRules.LongestPrefix(node)
if ok {
switch rule {
case PolicyWrite:
return true
default:
return false
}
}
return p.parent.OperatorWrite()
}
// ACLList checks if listing of ACLs is allowed
func (p *PolicyACL) ACLList() bool {
return p.parent.ACLList()
}
// ACLModify checks if modification of ACLs is allowed
func (p *PolicyACL) ACLModify() bool {
return p.parent.ACLModify()
}
// Snapshot checks if taking and restoring snapshots is allowed.
func (p *PolicyACL) Snapshot() bool {
return p.parent.Snapshot()
// No matching rule, use the parent.
return p.parent.SessionWrite(node)
}

459
acl/acl_test.go
View File

@ -35,16 +35,16 @@ func TestStaticACL(t *testing.T) {
t.Fatalf("expected static")
}
if !all.KeyRead("foobar") {
if all.ACLList() {
t.Fatalf("should not allow")
}
if all.ACLModify() {
t.Fatalf("should not allow")
}
if !all.AgentRead("foobar") {
t.Fatalf("should allow")
}
if !all.KeyWrite("foobar") {
t.Fatalf("should allow")
}
if !all.ServiceRead("foobar") {
t.Fatalf("should allow")
}
if !all.ServiceWrite("foobar") {
if !all.AgentWrite("foobar") {
t.Fatalf("should allow")
}
if !all.EventRead("foobar") {
@ -53,10 +53,10 @@ func TestStaticACL(t *testing.T) {
if !all.EventWrite("foobar") {
t.Fatalf("should allow")
}
if !all.PreparedQueryRead("foobar") {
if !all.KeyRead("foobar") {
t.Fatalf("should allow")
}
if !all.PreparedQueryWrite("foobar") {
if !all.KeyWrite("foobar") {
t.Fatalf("should allow")
}
if !all.KeyringRead() {
@ -65,32 +65,50 @@ func TestStaticACL(t *testing.T) {
if !all.KeyringWrite() {
t.Fatalf("should allow")
}
if !all.NodeRead("foobar") {
t.Fatalf("should allow")
}
if !all.NodeWrite("foobar") {
t.Fatalf("should allow")
}
if !all.OperatorRead() {
t.Fatalf("should allow")
}
if !all.OperatorWrite() {
t.Fatalf("should allow")
}
if all.ACLList() {
t.Fatalf("should not allow")
if !all.PreparedQueryRead("foobar") {
t.Fatalf("should allow")
}
if all.ACLModify() {
t.Fatalf("should not allow")
if !all.PreparedQueryWrite("foobar") {
t.Fatalf("should allow")
}
if !all.ServiceRead("foobar") {
t.Fatalf("should allow")
}
if !all.ServiceWrite("foobar") {
t.Fatalf("should allow")
}
if !all.SessionRead("foobar") {
t.Fatalf("should allow")
}
if !all.SessionWrite("foobar") {
t.Fatalf("should allow")
}
if all.Snapshot() {
t.Fatalf("should not allow")
}
if none.KeyRead("foobar") {
if none.ACLList() {
t.Fatalf("should not allow")
}
if none.KeyWrite("foobar") {
if none.ACLModify() {
t.Fatalf("should not allow")
}
if none.ServiceRead("foobar") {
if none.AgentRead("foobar") {
t.Fatalf("should not allow")
}
if none.ServiceWrite("foobar") {
if none.AgentWrite("foobar") {
t.Fatalf("should not allow")
}
if none.EventRead("foobar") {
@ -105,10 +123,10 @@ func TestStaticACL(t *testing.T) {
if none.EventWrite("") {
t.Fatalf("should not allow")
}
if none.PreparedQueryRead("foobar") {
if none.KeyRead("foobar") {
t.Fatalf("should not allow")
}
if none.PreparedQueryWrite("foobar") {
if none.KeyWrite("foobar") {
t.Fatalf("should not allow")
}
if none.KeyringRead() {
@ -117,32 +135,50 @@ func TestStaticACL(t *testing.T) {
if none.KeyringWrite() {
t.Fatalf("should not allow")
}
if none.NodeRead("foobar") {
t.Fatalf("should not allow")
}
if none.NodeWrite("foobar") {
t.Fatalf("should not allow")
}
if none.OperatorRead() {
t.Fatalf("should now allow")
}
if none.OperatorWrite() {
t.Fatalf("should not allow")
}
if none.ACLList() {
if none.PreparedQueryRead("foobar") {
t.Fatalf("should not allow")
}
if none.ACLModify() {
if none.PreparedQueryWrite("foobar") {
t.Fatalf("should not allow")
}
if none.ServiceRead("foobar") {
t.Fatalf("should not allow")
}
if none.ServiceWrite("foobar") {
t.Fatalf("should not allow")
}
if none.SessionRead("foobar") {
t.Fatalf("should not allow")
}
if none.SessionWrite("foobar") {
t.Fatalf("should not allow")
}
if none.Snapshot() {
t.Fatalf("should not allow")
}
if !manage.KeyRead("foobar") {
if !manage.ACLList() {
t.Fatalf("should allow")
}
if !manage.KeyWrite("foobar") {
if !manage.ACLModify() {
t.Fatalf("should allow")
}
if !manage.ServiceRead("foobar") {
if !manage.AgentRead("foobar") {
t.Fatalf("should allow")
}
if !manage.ServiceWrite("foobar") {
if !manage.AgentWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.EventRead("foobar") {
@ -151,10 +187,10 @@ func TestStaticACL(t *testing.T) {
if !manage.EventWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.PreparedQueryRead("foobar") {
if !manage.KeyRead("foobar") {
t.Fatalf("should allow")
}
if !manage.PreparedQueryWrite("foobar") {
if !manage.KeyWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.KeyringRead() {
@ -163,16 +199,34 @@ func TestStaticACL(t *testing.T) {
if !manage.KeyringWrite() {
t.Fatalf("should allow")
}
if !manage.NodeRead("foobar") {
t.Fatalf("should allow")
}
if !manage.NodeWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.OperatorRead() {
t.Fatalf("should allow")
}
if !manage.OperatorWrite() {
t.Fatalf("should allow")
}
if !manage.ACLList() {
if !manage.PreparedQueryRead("foobar") {
t.Fatalf("should allow")
}
if !manage.ACLModify() {
if !manage.PreparedQueryWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.ServiceRead("foobar") {
t.Fatalf("should allow")
}
if !manage.ServiceWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.SessionRead("foobar") {
t.Fatalf("should allow")
}
if !manage.SessionWrite("foobar") {
t.Fatalf("should allow")
}
if !manage.Snapshot() {
@ -183,6 +237,20 @@ func TestStaticACL(t *testing.T) {
func TestPolicyACL(t *testing.T) {
all := AllowAll()
policy := &Policy{
Events: []*EventPolicy{
&EventPolicy{
Event: "",
Policy: PolicyRead,
},
&EventPolicy{
Event: "foo",
Policy: PolicyWrite,
},
&EventPolicy{
Event: "bar",
Policy: PolicyDeny,
},
},
Keys: []*KeyPolicy{
&KeyPolicy{
Prefix: "foo/",
@ -201,38 +269,6 @@ func TestPolicyACL(t *testing.T) {
Policy: PolicyRead,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Name: "",
Policy: PolicyWrite,
},
&ServicePolicy{
Name: "foo",
Policy: PolicyRead,
},
&ServicePolicy{
Name: "bar",
Policy: PolicyDeny,
},
&ServicePolicy{
Name: "barfoo",
Policy: PolicyWrite,
},
},
Events: []*EventPolicy{
&EventPolicy{
Event: "",
Policy: PolicyRead,
},
&EventPolicy{
Event: "foo",
Policy: PolicyWrite,
},
&EventPolicy{
Event: "bar",
Policy: PolicyDeny,
},
},
PreparedQueries: []*PreparedQueryPolicy{
&PreparedQueryPolicy{
Prefix: "",
@ -251,6 +287,24 @@ func TestPolicyACL(t *testing.T) {
Policy: PolicyWrite,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Name: "",
Policy: PolicyWrite,
},
&ServicePolicy{
Name: "foo",
Policy: PolicyRead,
},
&ServicePolicy{
Name: "bar",
Policy: PolicyDeny,
},
&ServicePolicy{
Name: "barfoo",
Policy: PolicyWrite,
},
},
}
acl, err := New(all, policy)
if err != nil {
@ -369,16 +423,6 @@ func TestPolicyACL_Parent(t *testing.T) {
Policy: PolicyRead,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Name: "other",
Policy: PolicyWrite,
},
&ServicePolicy{
Name: "foo",
Policy: PolicyRead,
},
},
PreparedQueries: []*PreparedQueryPolicy{
&PreparedQueryPolicy{
Prefix: "other",
@ -389,6 +433,16 @@ func TestPolicyACL_Parent(t *testing.T) {
Policy: PolicyRead,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Name: "other",
Policy: PolicyWrite,
},
&ServicePolicy{
Name: "foo",
Policy: PolicyRead,
},
},
}
root, err := New(deny, policyRoot)
if err != nil {
@ -410,18 +464,18 @@ func TestPolicyACL_Parent(t *testing.T) {
Policy: PolicyRead,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Name: "bar",
Policy: PolicyDeny,
},
},
PreparedQueries: []*PreparedQueryPolicy{
&PreparedQueryPolicy{
Prefix: "bar",
Policy: PolicyDeny,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Name: "bar",
Policy: PolicyDeny,
},
},
}
acl, err := New(root, policy)
if err != nil {
@ -509,6 +563,89 @@ func TestPolicyACL_Parent(t *testing.T) {
}
}
func TestPolicyACL_Agent(t *testing.T) {
deny := DenyAll()
policyRoot := &Policy{
Agents: []*AgentPolicy{
&AgentPolicy{
Node: "root-nope",
Policy: PolicyDeny,
},
&AgentPolicy{
Node: "root-ro",
Policy: PolicyRead,
},
&AgentPolicy{
Node: "root-rw",
Policy: PolicyWrite,
},
&AgentPolicy{
Node: "override",
Policy: PolicyDeny,
},
},
}
root, err := New(deny, policyRoot)
if err != nil {
t.Fatalf("err: %v", err)
}
policy := &Policy{
Agents: []*AgentPolicy{
&AgentPolicy{
Node: "child-nope",
Policy: PolicyDeny,
},
&AgentPolicy{
Node: "child-ro",
Policy: PolicyRead,
},
&AgentPolicy{
Node: "child-rw",
Policy: PolicyWrite,
},
&AgentPolicy{
Node: "override",
Policy: PolicyWrite,
},
},
}
acl, err := New(root, policy)
if err != nil {
t.Fatalf("err: %v", err)
}
type agentcase struct {
inp string
read bool
write bool
}
cases := []agentcase{
{"nope", false, false},
{"root-nope", false, false},
{"root-ro", true, false},
{"root-rw", true, true},
{"root-nope-prefix", false, false},
{"root-ro-prefix", true, false},
{"root-rw-prefix", true, true},
{"child-nope", false, false},
{"child-ro", true, false},
{"child-rw", true, true},
{"child-nope-prefix", false, false},
{"child-ro-prefix", true, false},
{"child-rw-prefix", true, true},
{"override", true, true},
}
for _, c := range cases {
if c.read != acl.AgentRead(c.inp) {
t.Fatalf("Read fail: %#v", c)
}
if c.write != acl.AgentWrite(c.inp) {
t.Fatalf("Write fail: %#v", c)
}
}
}
func TestPolicyACL_Keyring(t *testing.T) {
type keyringcase struct {
inp string
@ -560,3 +697,169 @@ func TestPolicyACL_Operator(t *testing.T) {
}
}
}
func TestPolicyACL_Node(t *testing.T) {
deny := DenyAll()
policyRoot := &Policy{
Nodes: []*NodePolicy{
&NodePolicy{
Name: "root-nope",
Policy: PolicyDeny,
},
&NodePolicy{
Name: "root-ro",
Policy: PolicyRead,
},
&NodePolicy{
Name: "root-rw",
Policy: PolicyWrite,
},
&NodePolicy{
Name: "override",
Policy: PolicyDeny,
},
},
}
root, err := New(deny, policyRoot)
if err != nil {
t.Fatalf("err: %v", err)
}
policy := &Policy{
Nodes: []*NodePolicy{
&NodePolicy{
Name: "child-nope",
Policy: PolicyDeny,
},
&NodePolicy{
Name: "child-ro",
Policy: PolicyRead,
},
&NodePolicy{
Name: "child-rw",
Policy: PolicyWrite,
},
&NodePolicy{
Name: "override",
Policy: PolicyWrite,
},
},
}
acl, err := New(root, policy)
if err != nil {
t.Fatalf("err: %v", err)
}
type nodecase struct {
inp string
read bool
write bool
}
cases := []nodecase{
{"nope", false, false},
{"root-nope", false, false},
{"root-ro", true, false},
{"root-rw", true, true},
{"root-nope-prefix", false, false},
{"root-ro-prefix", true, false},
{"root-rw-prefix", true, true},
{"child-nope", false, false},
{"child-ro", true, false},
{"child-rw", true, true},
{"child-nope-prefix", false, false},
{"child-ro-prefix", true, false},
{"child-rw-prefix", true, true},
{"override", true, true},
}
for _, c := range cases {
if c.read != acl.NodeRead(c.inp) {
t.Fatalf("Read fail: %#v", c)
}
if c.write != acl.NodeWrite(c.inp) {
t.Fatalf("Write fail: %#v", c)
}
}
}
func TestPolicyACL_Session(t *testing.T) {
deny := DenyAll()
policyRoot := &Policy{
Sessions: []*SessionPolicy{
&SessionPolicy{
Node: "root-nope",
Policy: PolicyDeny,
},
&SessionPolicy{
Node: "root-ro",
Policy: PolicyRead,
},
&SessionPolicy{
Node: "root-rw",
Policy: PolicyWrite,
},
&SessionPolicy{
Node: "override",
Policy: PolicyDeny,
},
},
}
root, err := New(deny, policyRoot)
if err != nil {
t.Fatalf("err: %v", err)
}
policy := &Policy{
Sessions: []*SessionPolicy{
&SessionPolicy{
Node: "child-nope",
Policy: PolicyDeny,
},
&SessionPolicy{
Node: "child-ro",
Policy: PolicyRead,
},
&SessionPolicy{
Node: "child-rw",
Policy: PolicyWrite,
},
&SessionPolicy{
Node: "override",
Policy: PolicyWrite,
},
},
}
acl, err := New(root, policy)
if err != nil {
t.Fatalf("err: %v", err)
}
type sessioncase struct {
inp string
read bool
write bool
}
cases := []sessioncase{
{"nope", false, false},
{"root-nope", false, false},
{"root-ro", true, false},
{"root-rw", true, true},
{"root-nope-prefix", false, false},
{"root-ro-prefix", true, false},
{"root-rw-prefix", true, true},
{"child-nope", false, false},
{"child-ro", true, false},
{"child-rw", true, true},
{"child-nope-prefix", false, false},
{"child-ro-prefix", true, false},
{"child-rw-prefix", true, true},
{"override", true, true},
}
for _, c := range cases {
if c.read != acl.SessionRead(c.inp) {
t.Fatalf("Read fail: %#v", c)
}
if c.write != acl.SessionWrite(c.inp) {
t.Fatalf("Write fail: %#v", c)
}
}
}

66
acl/policy.go
View File

@ -16,14 +16,28 @@ const (
// an ACL configuration.
type Policy struct {
ID string `hcl:"-"`
Agents []*AgentPolicy `hcl:"agent,expand"`
Keys []*KeyPolicy `hcl:"key,expand"`
Nodes []*NodePolicy `hcl:"node,expand"`
Services []*ServicePolicy `hcl:"service,expand"`
Sessions []*SessionPolicy `hcl:"session,expand"`
Events []*EventPolicy `hcl:"event,expand"`
PreparedQueries []*PreparedQueryPolicy `hcl:"query,expand"`
Keyring string `hcl:"keyring"`
Operator string `hcl:"operator"`
}
// AgentPolicy represents a policy for working with agent endpoints on nodes
// with specific name prefixes.
type AgentPolicy struct {
Node string `hcl:",key"`
Policy string
}
func (a *AgentPolicy) GoString() string {
return fmt.Sprintf("%#v", *a)
}
// KeyPolicy represents a policy for a key
type KeyPolicy struct {
Prefix string `hcl:",key"`
@ -34,14 +48,35 @@ func (k *KeyPolicy) GoString() string {
return fmt.Sprintf("%#v", *k)
}
// NodePolicy represents a policy for a node
type NodePolicy struct {
Name string `hcl:",key"`
Policy string
}
func (n *NodePolicy) GoString() string {
return fmt.Sprintf("%#v", *n)
}
// ServicePolicy represents a policy for a service
type ServicePolicy struct {
Name string `hcl:",key"`
Policy string
}
func (k *ServicePolicy) GoString() string {
return fmt.Sprintf("%#v", *k)
func (s *ServicePolicy) GoString() string {
return fmt.Sprintf("%#v", *s)
}
// SessionPolicy represents a policy for making sessions tied to specific node
// name prefixes.
type SessionPolicy struct {
Node string `hcl:",key"`
Policy string
}
func (s *SessionPolicy) GoString() string {
return fmt.Sprintf("%#v", *s)
}
// EventPolicy represents a user event policy.
@ -60,8 +95,8 @@ type PreparedQueryPolicy struct {
Policy string
}
func (e *PreparedQueryPolicy) GoString() string {
return fmt.Sprintf("%#v", *e)
func (p *PreparedQueryPolicy) GoString() string {
return fmt.Sprintf("%#v", *p)
}
// isPolicyValid makes sure the given string matches one of the valid policies.
@ -93,6 +128,13 @@ func Parse(rules string) (*Policy, error) {
return nil, fmt.Errorf("Failed to parse ACL rules: %v", err)
}
// Validate the agent policy
for _, ap := range p.Agents {
if !isPolicyValid(ap.Policy) {
return nil, fmt.Errorf("Invalid agent policy: %#v", ap)
}
}
// Validate the key policy
for _, kp := range p.Keys {
if !isPolicyValid(kp.Policy) {
@ -100,13 +142,27 @@ func Parse(rules string) (*Policy, error) {
}
}
// Validate the service policy
// Validate the node policies
for _, np := range p.Nodes {
if !isPolicyValid(np.Policy) {
return nil, fmt.Errorf("Invalid node policy: %#v", np)
}
}
// Validate the service policies
for _, sp := range p.Services {
if !isPolicyValid(sp.Policy) {
return nil, fmt.Errorf("Invalid service policy: %#v", sp)
}
}
// Validate the session policies
for _, sp := range p.Sessions {
if !isPolicyValid(sp.Policy) {
return nil, fmt.Errorf("Invalid session policy: %#v", sp)
}
}
// Validate the user event policies
for _, ep := range p.Events {
if !isPolicyValid(ep.Policy) {

233
acl/policy_test.go
View File

@ -8,6 +8,21 @@ import (
func TestACLPolicy_Parse_HCL(t *testing.T) {
inp := `
agent "foo" {
policy = "read"
}
agent "bar" {
policy = "write"
}
event "" {
policy = "read"
}
event "foo" {
policy = "write"
}
event "bar" {
policy = "deny"
}
key "" {
policy = "read"
}
@ -20,19 +35,27 @@ key "foo/bar/" {
key "foo/bar/baz" {
policy = "deny"
}
keyring = "deny"
node "" {
policy = "read"
}
node "foo" {
policy = "write"
}
node "bar" {
policy = "deny"
}
operator = "deny"
service "" {
policy = "write"
}
service "foo" {
policy = "read"
}
event "" {
policy = "read"
}
event "foo" {
session "foo" {
policy = "write"
}
event "bar" {
session "bar" {
policy = "deny"
}
query "" {
@ -44,10 +67,33 @@ query "foo" {
query "bar" {
policy = "deny"
}
keyring = "deny"
operator = "deny"
`
exp := &Policy{
Agents: []*AgentPolicy{
&AgentPolicy{
Node: "foo",
Policy: PolicyRead,
},
&AgentPolicy{
Node: "bar",
Policy: PolicyWrite,
},
},
Events: []*EventPolicy{
&EventPolicy{
Event: "",
Policy: PolicyRead,
},
&EventPolicy{
Event: "foo",
Policy: PolicyWrite,
},
&EventPolicy{
Event: "bar",
Policy: PolicyDeny,
},
},
Keyring: PolicyDeny,
Keys: []*KeyPolicy{
&KeyPolicy{
Prefix: "",
@ -66,30 +112,21 @@ operator = "deny"
Policy: PolicyDeny,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Nodes: []*NodePolicy{
&NodePolicy{
Name: "",
Policy: PolicyWrite,
Policy: PolicyRead,
},
&ServicePolicy{
&NodePolicy{
Name: "foo",
Policy: PolicyRead,
},
},
Events: []*EventPolicy{
&EventPolicy{
Event: "",
Policy: PolicyRead,
},
&EventPolicy{
Event: "foo",
Policy: PolicyWrite,
},
&EventPolicy{
Event: "bar",
&NodePolicy{
Name: "bar",
Policy: PolicyDeny,
},
},
Operator: PolicyDeny,
PreparedQueries: []*PreparedQueryPolicy{
&PreparedQueryPolicy{
Prefix: "",
@ -104,8 +141,26 @@ operator = "deny"
Policy: PolicyDeny,
},
},
Keyring: PolicyDeny,
Operator: PolicyDeny,
Services: []*ServicePolicy{
&ServicePolicy{
Name: "",
Policy: PolicyWrite,
},
&ServicePolicy{
Name: "foo",
Policy: PolicyRead,
},
},
Sessions: []*SessionPolicy{
&SessionPolicy{
Node: "foo",
Policy: PolicyWrite,
},
&SessionPolicy{
Node: "bar",
Policy: PolicyDeny,
},
},
}
out, err := Parse(inp)
@ -120,6 +175,25 @@ operator = "deny"
func TestACLPolicy_Parse_JSON(t *testing.T) {
inp := `{
"agent": {
"foo": {
"policy": "write"
},
"bar": {
"policy": "deny"
}
},
"event": {
"": {
"policy": "read"
},
"foo": {
"policy": "write"
},
"bar": {
"policy": "deny"
}
},
"key": {
"": {
"policy": "read"
@ -134,15 +208,8 @@ func TestACLPolicy_Parse_JSON(t *testing.T) {
"policy": "deny"
}
},
"service": {
"": {
"policy": "write"
},
"foo": {
"policy": "read"
}
},
"event": {
"keyring": "deny",
"node": {
"": {
"policy": "read"
},
@ -153,6 +220,7 @@ func TestACLPolicy_Parse_JSON(t *testing.T) {
"policy": "deny"
}
},
"operator": "deny",
"query": {
"": {
"policy": "read"
@ -164,10 +232,49 @@ func TestACLPolicy_Parse_JSON(t *testing.T) {
"policy": "deny"
}
},
"keyring": "deny",
"operator": "deny"
"service": {
"": {
"policy": "write"
},
"foo": {
"policy": "read"
}
},
"session": {
"foo": {
"policy": "write"
},
"bar": {
"policy": "deny"
}
}
}`
exp := &Policy{
Agents: []*AgentPolicy{
&AgentPolicy{
Node: "foo",
Policy: PolicyWrite,
},
&AgentPolicy{
Node: "bar",
Policy: PolicyDeny,
},
},
Events: []*EventPolicy{
&EventPolicy{
Event: "",
Policy: PolicyRead,
},
&EventPolicy{
Event: "foo",
Policy: PolicyWrite,
},
&EventPolicy{
Event: "bar",
Policy: PolicyDeny,
},
},
Keyring: PolicyDeny,
Keys: []*KeyPolicy{
&KeyPolicy{
Prefix: "",
@ -186,30 +293,21 @@ func TestACLPolicy_Parse_JSON(t *testing.T) {
Policy: PolicyDeny,
},
},
Services: []*ServicePolicy{
&ServicePolicy{
Nodes: []*NodePolicy{
&NodePolicy{
Name: "",
Policy: PolicyWrite,
Policy: PolicyRead,
},
&ServicePolicy{
&NodePolicy{
Name: "foo",
Policy: PolicyRead,
},
},
Events: []*EventPolicy{
&EventPolicy{
Event: "",
Policy: PolicyRead,
},
&EventPolicy{
Event: "foo",
Policy: PolicyWrite,
},
&EventPolicy{
Event: "bar",
&NodePolicy{
Name: "bar",
Policy: PolicyDeny,
},
},
Operator: PolicyDeny,
PreparedQueries: []*PreparedQueryPolicy{
&PreparedQueryPolicy{
Prefix: "",
@ -224,8 +322,26 @@ func TestACLPolicy_Parse_JSON(t *testing.T) {
Policy: PolicyDeny,
},
},
Keyring: PolicyDeny,
Operator: PolicyDeny,
Services: []*ServicePolicy{
&ServicePolicy{
Name: "",
Policy: PolicyWrite,
},
&ServicePolicy{
Name: "foo",
Policy: PolicyRead,
},
},
Sessions: []*SessionPolicy{
&SessionPolicy{
Node: "foo",
Policy: PolicyWrite,
},
&SessionPolicy{
Node: "bar",
Policy: PolicyDeny,
},
},
}
out, err := Parse(inp)
@ -276,12 +392,15 @@ operator = ""
func TestACLPolicy_Bad_Policy(t *testing.T) {
cases := []string{
`key "" { policy = "nope" }`,
`service "" { policy = "nope" }`,
`agent "" { policy = "nope" }`,
`event "" { policy = "nope" }`,
`query "" { policy = "nope" }`,
`key "" { policy = "nope" }`,
`keyring = "nope"`,
`node "" { policy = "nope" }`,
`operator = "nope"`,
`query "" { policy = "nope" }`,
`service "" { policy = "nope" }`,
`session "" { policy = "nope" }`,
}
for _, c := range cases {
_, err := Parse(c)

10
api/api.go
View File

@ -74,6 +74,11 @@ type QueryOptions struct {
// that node. Setting this to "_agent" will use the agent's node
// for the sort.
Near string
// NodeMeta is used to filter results by nodes with the given
// metadata key/value pairs. Currently, only one key/value pair can
// be provided for filtering.
NodeMeta map[string]string
}
// WriteOptions are used to parameterize a write
@ -386,6 +391,11 @@ func (r *request) setQueryOptions(q *QueryOptions) {
if q.Near != "" {
r.params.Set("near", q.Near)
}
if len(q.NodeMeta) > 0 {
for key, value := range q.NodeMeta {
r.params.Add("node-meta", key+":"+value)
}
}
}
// durToMsec converts a duration to a millisecond specified string. If the

3
api/catalog.go
View File

@ -4,12 +4,14 @@ type Node struct {
Node string
Address string
TaggedAddresses map[string]string
Meta map[string]string
}
type CatalogService struct {
Node string
Address string
TaggedAddresses map[string]string
NodeMeta map[string]string
ServiceID string
ServiceName string
ServiceAddress string
@ -29,6 +31,7 @@ type CatalogRegistration struct {
Node string
Address string
TaggedAddresses map[string]string
NodeMeta map[string]string
Datacenter string
Service *AgentService
Check *AgentCheck

113
api/catalog_test.go
View File

@ -60,6 +60,64 @@ func TestCatalog_Nodes(t *testing.T) {
})
}
func TestCatalog_Nodes_MetaFilter(t *testing.T) {
meta := map[string]string{"somekey": "somevalue"}
c, s := makeClientWithConfig(t, nil, func(conf *testutil.TestServerConfig) {
conf.NodeMeta = meta
})
defer s.Stop()
catalog := c.Catalog()
// Make sure we get the node back when filtering by its metadata
testutil.WaitForResult(func() (bool, error) {
nodes, meta, err := catalog.Nodes(&QueryOptions{NodeMeta: meta})
if err != nil {
return false, err
}
if meta.LastIndex == 0 {
return false, fmt.Errorf("Bad: %v", meta)
}
if len(nodes) == 0 {
return false, fmt.Errorf("Bad: %v", nodes)
}
if _, ok := nodes[0].TaggedAddresses["wan"]; !ok {
return false, fmt.Errorf("Bad: %v", nodes[0])
}
if v, ok := nodes[0].Meta["somekey"]; !ok || v != "somevalue" {
return false, fmt.Errorf("Bad: %v", nodes[0].Meta)
}
return true, nil
}, func(err error) {
t.Fatalf("err: %s", err)
})
// Get nothing back when we use an invalid filter
testutil.WaitForResult(func() (bool, error) {
nodes, meta, err := catalog.Nodes(&QueryOptions{NodeMeta: map[string]string{"nope": "nope"}})
if err != nil {
return false, err
}
if meta.LastIndex == 0 {
return false, fmt.Errorf("Bad: %v", meta)
}
if len(nodes) != 0 {
return false, fmt.Errorf("Bad: %v", nodes)
}
return true, nil
}, func(err error) {
t.Fatalf("err: %s", err)
})
}
func TestCatalog_Services(t *testing.T) {
t.Parallel()
c, s := makeClient(t)
@ -87,6 +145,56 @@ func TestCatalog_Services(t *testing.T) {
})
}
func TestCatalog_Services_NodeMetaFilter(t *testing.T) {
meta := map[string]string{"somekey": "somevalue"}
c, s := makeClientWithConfig(t, nil, func(conf *testutil.TestServerConfig) {
conf.NodeMeta = meta
})
defer s.Stop()
catalog := c.Catalog()
// Make sure we get the service back when filtering by the node's metadata
testutil.WaitForResult(func() (bool, error) {
services, meta, err := catalog.Services(&QueryOptions{NodeMeta: meta})
if err != nil {
return false, err
}
if meta.LastIndex == 0 {
return false, fmt.Errorf("Bad: %v", meta)
}
if len(services) == 0 {
return false, fmt.Errorf("Bad: %v", services)
}
return true, nil
}, func(err error) {
t.Fatalf("err: %s", err)
})
// Get nothing back when using an invalid filter
testutil.WaitForResult(func() (bool, error) {
services, meta, err := catalog.Services(&QueryOptions{NodeMeta: map[string]string{"nope": "nope"}})
if err != nil {
return false, err
}
if meta.LastIndex == 0 {
return false, fmt.Errorf("Bad: %v", meta)
}
if len(services) != 0 {
return false, fmt.Errorf("Bad: %v", services)
}
return true, nil
}, func(err error) {
t.Fatalf("err: %s", err)
})
}
func TestCatalog_Service(t *testing.T) {
t.Parallel()
c, s := makeClient(t)
@ -173,6 +281,7 @@ func TestCatalog_Registration(t *testing.T) {
Datacenter: "dc1",
Node: "foobar",
Address: "192.168.10.10",
NodeMeta: map[string]string{"somekey": "somevalue"},
Service: service,
Check: check,
}
@ -200,6 +309,10 @@ func TestCatalog_Registration(t *testing.T) {
return false, fmt.Errorf("missing checkid service:redis1")
}
if v, ok := node.Node.Meta["somekey"]; !ok || v != "somevalue" {
return false, fmt.Errorf("missing node meta pair somekey:somevalue")
}
return true, nil
}, func(err error) {
t.Fatalf("err: %s", err)

1
api/prepared_query_test.go
View File

@ -45,6 +45,7 @@ func TestPreparedQuery(t *testing.T) {
// Create a simple prepared query.
def := &PreparedQueryDefinition{
Name: "test",
Service: ServiceQuery{
Service: "redis",
},

452
command/agent/acl.go Normal file
View File

@ -0,0 +1,452 @@
package agent
import (
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/armon/go-metrics"
"github.com/hashicorp/consul/acl"
"github.com/hashicorp/consul/consul/structs"
"github.com/hashicorp/consul/types"
"github.com/hashicorp/golang-lru"
"github.com/hashicorp/serf/serf"
)
// There's enough behavior difference with client-side ACLs that we've
// intentionally kept this code separate from the server-side ACL code in
// consul/acl.go. We may refactor some of the caching logic in the future,
// but for now we are developing this separately to see how things shake out.
// These must be kept in sync with the constants in consul/acl.go.
const (
// aclNotFound indicates there is no matching ACL.
aclNotFound = "ACL not found"
// rootDenied is returned when attempting to resolve a root ACL.
rootDenied = "Cannot resolve root ACL"
// permissionDenied is returned when an ACL based rejection happens.
permissionDenied = "Permission denied"
// aclDisabled is returned when ACL changes are not permitted since they
// are disabled.
aclDisabled = "ACL support disabled"
// anonymousToken is the token ID we re-write to if there is no token ID
// provided.
anonymousToken = "anonymous"
// Maximum number of cached ACL entries.
aclCacheSize = 10 * 1024
)
var (
permissionDeniedErr = errors.New(permissionDenied)
)
// aclCacheEntry is used to cache ACL tokens.
type aclCacheEntry struct {
// ACL is the cached ACL.
ACL acl.ACL
// Expires is set based on the TTL for the ACL.
Expires time.Time
// ETag is used as an optimization when fetching ACLs from servers to
// avoid transmitting data back when the agent has a good copy, which is
// usually the case when refreshing a TTL.
ETag string
}
// aclManager is used by the agent to keep track of state related to ACLs,
// including caching tokens from the servers. This has some internal state that
// we don't want to dump into the agent itself.
type aclManager struct {
// acls is a cache mapping ACL tokens to compiled policies.
acls *lru.TwoQueueCache
// master is the ACL to use when the agent master token is supplied.
// This may be nil if that option isn't set in the agent config.
master acl.ACL
// down is the ACL to use when the servers are down. This may be nil
// which means to try and use the cached policy if there is one (or
// deny if there isn't a policy in the cache).
down acl.ACL
// disabled is used to keep track of feedback from the servers that ACLs
// are disabled. If the manager discovers that ACLs are disabled, this
// will be set to the next time we should check to see if they have been
// enabled. This helps cut useless traffic, but allows us to turn on ACL
// support at the servers without having to restart the whole cluster.
disabled time.Time
disabledLock sync.RWMutex
}
// newACLManager returns an ACL manager based on the given config.
func newACLManager(config *Config) (*aclManager, error) {
// Set up the cache from ID to ACL (we don't cache policies like the
// servers; only one level).
acls, err := lru.New2Q(aclCacheSize)
if err != nil {
return nil, err
}
// If an agent master token is configured, build a policy and ACL for
// it, otherwise leave it nil.
var master acl.ACL
if len(config.ACLAgentMasterToken) > 0 {
policy := &acl.Policy{
Agents: []*acl.AgentPolicy{
&acl.AgentPolicy{
Node: config.NodeName,
Policy: acl.PolicyWrite,
},
},
}
acl, err := acl.New(acl.DenyAll(), policy)
if err != nil {
return nil, err
}
master = acl
}
var down acl.ACL
switch config.ACLDownPolicy {
case "allow":
down = acl.AllowAll()
case "deny":
down = acl.DenyAll()
case "extend-cache":
// Leave the down policy as nil to signal this.
default:
return nil, fmt.Errorf("invalid ACL down policy %q", config.ACLDownPolicy)
}
// Give back a manager.
return &aclManager{
acls: acls,
master: master,
down: down,
}, nil
}
// isDisabled returns true if the manager has discovered that ACLs are disabled
// on the servers.
func (m *aclManager) isDisabled() bool {
m.disabledLock.RLock()
defer m.disabledLock.RUnlock()
return time.Now().Before(m.disabled)
}
// lookupACL attempts to locate the compiled policy associated with the given
// token. The agent may be used to perform RPC calls to the servers to fetch
// policies that aren't in the cache.
func (m *aclManager) lookupACL(agent *Agent, id string) (acl.ACL, error) {
// Handle some special cases for the ID.
if len(id) == 0 {
id = anonymousToken
} else if acl.RootACL(id) != nil {
return nil, errors.New(rootDenied)
} else if m.master != nil && id == agent.config.ACLAgentMasterToken {
return m.master, nil
}
// Try the cache first.
var cached *aclCacheEntry
if raw, ok := m.acls.Get(id); ok {
cached = raw.(*aclCacheEntry)
}
if cached != nil && time.Now().Before(cached.Expires) {
metrics.IncrCounter([]string{"consul", "acl", "cache_hit"}, 1)
return cached.ACL, nil
} else {
metrics.IncrCounter([]string{"consul", "acl", "cache_miss"}, 1)
}
// At this point we might have a stale cached ACL, or none at all, so
// try to contact the servers.
args := structs.ACLPolicyRequest{
Datacenter: agent.config.Datacenter,
ACL: id,
}
if cached != nil {
args.ETag = cached.ETag
}
var reply structs.ACLPolicy
err := agent.RPC(agent.getEndpoint("ACL")+".GetPolicy", &args, &reply)
if err != nil {
if strings.Contains(err.Error(), aclDisabled) {
agent.logger.Printf("[DEBUG] agent: ACLs disabled on servers, will check again after %s", agent.config.ACLDisabledTTL)
m.disabledLock.Lock()
m.disabled = time.Now().Add(agent.config.ACLDisabledTTL)
m.disabledLock.Unlock()
return nil, nil
} else if strings.Contains(err.Error(), aclNotFound) {
return nil, errors.New(aclNotFound)
} else {
agent.logger.Printf("[DEBUG] agent: Failed to get policy for ACL from servers: %v", err)
if m.down != nil {
return m.down, nil
} else if cached != nil {
return cached.ACL, nil
} else {
return acl.DenyAll(), nil
}
}
}
// Use the old cached compiled ACL if we can, otherwise compile it and
// resolve any parents.
var compiled acl.ACL
if cached != nil && cached.ETag == reply.ETag {
compiled = cached.ACL
} else {
parent := acl.RootACL(reply.Parent)
if parent == nil {
parent, err = m.lookupACL(agent, reply.Parent)
if err != nil {
return nil, err
}
}
acl, err := acl.New(parent, reply.Policy)
if err != nil {
return nil, err
}
compiled = acl
}
// Update the cache.
cached = &aclCacheEntry{
ACL: compiled,
ETag: reply.ETag,
}
if reply.TTL > 0 {
cached.Expires = time.Now().Add(reply.TTL)
}
m.acls.Add(id, cached)
return compiled, nil
}
// resolveToken is the primary interface used by ACL-checkers in the agent
// endpoints, which is the one place where we do some ACL enforcement on
// clients. Some of the enforcement is normative (e.g. self and monitor)
// and some is informative (e.g. catalog and health).
func (a *Agent) resolveToken(id string) (acl.ACL, error) {
// Disable ACLs if version 8 enforcement isn't enabled.
if !(*a.config.ACLEnforceVersion8) {
return nil, nil
}
// Bail if the ACL manager is disabled. This happens if it gets feedback
// from the servers that ACLs are disabled.
if a.acls.isDisabled() {
return nil, nil
}
// This will look in the cache and fetch from the servers if necessary.
return a.acls.lookupACL(a, id)
}
// vetServiceRegister makes sure the service registration action is allowed by
// the given token.
func (a *Agent) vetServiceRegister(token string, service *structs.NodeService) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Vet the service itself.
if !acl.ServiceWrite(service.Service) {
return permissionDeniedErr
}
// Vet any service that might be getting overwritten.
services := a.state.Services()
if existing, ok := services[service.ID]; ok {
if !acl.ServiceWrite(existing.Service) {
return permissionDeniedErr
}
}
return nil
}
// vetServiceUpdate makes sure the service update action is allowed by the given
// token.
func (a *Agent) vetServiceUpdate(token string, serviceID string) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Vet any changes based on the existing services's info.
services := a.state.Services()
if existing, ok := services[serviceID]; ok {
if !acl.ServiceWrite(existing.Service) {
return permissionDeniedErr
}
} else {
return fmt.Errorf("Unknown service %q", serviceID)
}
return nil
}
// vetCheckRegister makes sure the check registration action is allowed by the
// given token.
func (a *Agent) vetCheckRegister(token string, check *structs.HealthCheck) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Vet the check itself.
if len(check.ServiceName) > 0 {
if !acl.ServiceWrite(check.ServiceName) {
return permissionDeniedErr
}
} else {
if !acl.NodeWrite(a.config.NodeName) {
return permissionDeniedErr
}
}
// Vet any check that might be getting overwritten.
checks := a.state.Checks()
if existing, ok := checks[check.CheckID]; ok {
if len(existing.ServiceName) > 0 {
if !acl.ServiceWrite(existing.ServiceName) {
return permissionDeniedErr
}
} else {
if !acl.NodeWrite(a.config.NodeName) {
return permissionDeniedErr
}
}
}
return nil
}
// vetCheckUpdate makes sure that a check update is allowed by the given token.
func (a *Agent) vetCheckUpdate(token string, checkID types.CheckID) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Vet any changes based on the existing check's info.
checks := a.state.Checks()
if existing, ok := checks[checkID]; ok {
if len(existing.ServiceName) > 0 {
if !acl.ServiceWrite(existing.ServiceName) {
return permissionDeniedErr
}
} else {
if !acl.NodeWrite(a.config.NodeName) {
return permissionDeniedErr
}
}
} else {
return fmt.Errorf("Unknown check %q", checkID)
}
return nil
}
// filterMembers redacts members that the token doesn't have access to.
func (a *Agent) filterMembers(token string, members *[]serf.Member) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Filter out members based on the node policy.
m := *members
for i := 0; i < len(m); i++ {
node := m[i].Name
if acl.NodeRead(node) {
continue
}
a.logger.Printf("[DEBUG] agent: dropping node %q from result due to ACLs", node)
m = append(m[:i], m[i+1:]...)
i--
}
*members = m
return nil
}
// filterServices redacts services that the token doesn't have access to.
func (a *Agent) filterServices(token string, services *map[string]*structs.NodeService) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Filter out services based on the service policy.
for id, service := range *services {
if acl.ServiceRead(service.Service) {
continue
}
a.logger.Printf("[DEBUG] agent: dropping service %q from result due to ACLs", id)
delete(*services, id)
}
return nil
}
// filterChecks redacts checks that the token doesn't have access to.
func (a *Agent) filterChecks(token string, checks *map[types.CheckID]*structs.HealthCheck) error {
// Resolve the token and bail if ACLs aren't enabled.
acl, err := a.resolveToken(token)
if err != nil {
return err
}
if acl == nil {
return nil
}
// Filter out checks based on the node or service policy.
for id, check := range *checks {
if len(check.ServiceName) > 0 {
if acl.ServiceRead(check.ServiceName) {
continue
}
} else {
if acl.NodeRead(a.config.NodeName) {
continue
}
}
a.logger.Printf("[DEBUG] agent: dropping check %q from result due to ACLs", id)
delete(*checks, id)
}
return nil
}

4
command/agent/acl_endpoint.go
View File

@ -13,8 +13,8 @@ type aclCreateResponse struct {
ID string
}
// aclDisabled handles if ACL datacenter is not configured
func aclDisabled(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// ACLDisabled handles if ACL datacenter is not configured
func ACLDisabled(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
resp.WriteHeader(401)
resp.Write([]byte("ACL support disabled"))
return nil, nil

863
command/agent/acl_test.go Normal file
View File

@ -0,0 +1,863 @@
package agent
import (
"errors"
"fmt"
"io/ioutil"
"os"
"strings"
"testing"
"time"
rawacl "github.com/hashicorp/consul/acl"
"github.com/hashicorp/consul/consul/structs"
"github.com/hashicorp/consul/testutil"
"github.com/hashicorp/consul/types"
"github.com/hashicorp/serf/serf"
)
func TestACL_Bad_Config(t *testing.T) {
config := nextConfig()
config.ACLDownPolicy = "nope"
var err error
config.DataDir, err = ioutil.TempDir("", "agent")
if err != nil {
t.Fatalf("err: %v", err)
}
defer os.RemoveAll(config.DataDir)
_, err = Create(config, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "invalid ACL down policy") {
t.Fatalf("err: %v", err)
}
}
type MockServer struct {
getPolicyFn func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error
}
func (m *MockServer) GetPolicy(args *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
if m.getPolicyFn != nil {
return m.getPolicyFn(args, reply)
} else {
return fmt.Errorf("should not have called GetPolicy")
}
}
func TestACL_Version8(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(false)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// With version 8 enforcement off, this should not get called.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
t.Fatalf("should not have called to server")
return nil
}
if token, err := agent.resolveToken("nope"); token != nil || err != nil {
t.Fatalf("bad: %v err: %v", token, err)
}
}
func TestACL_Disabled(t *testing.T) {
config := nextConfig()
config.ACLDisabledTTL = 10 * time.Millisecond
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Fetch a token without ACLs enabled and make sure the manager sees it.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
return errors.New(aclDisabled)
}
if agent.acls.isDisabled() {
t.Fatalf("should not be disabled yet")
}
if token, err := agent.resolveToken("nope"); token != nil || err != nil {
t.Fatalf("bad: %v err: %v", token, err)
}
if !agent.acls.isDisabled() {
t.Fatalf("should be disabled")
}
// Now turn on ACLs and check right away, it should still think ACLs are
// disabled since we don't check again right away.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
return errors.New(aclNotFound)
}
if token, err := agent.resolveToken("nope"); token != nil || err != nil {
t.Fatalf("bad: %v err: %v", token, err)
}
if !agent.acls.isDisabled() {
t.Fatalf("should be disabled")
}
// Wait the waiting period and make sure it checks again. Do a few tries
// to make sure we don't think it's disabled.
time.Sleep(2 * config.ACLDisabledTTL)
for i := 0; i < 10; i++ {
_, err := agent.resolveToken("nope")
if err == nil || !strings.Contains(err.Error(), aclNotFound) {
t.Fatalf("err: %v", err)
}
if agent.acls.isDisabled() {
t.Fatalf("should not be disabled")
}
}
}
func TestACL_Special_IDs(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
config.ACLAgentMasterToken = "towel"
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// An empty ID should get mapped to the anonymous token.
m.getPolicyFn = func(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
if req.ACL != "anonymous" {
t.Fatalf("bad: %#v", *req)
}
return errors.New(aclNotFound)
}
_, err := agent.resolveToken("")
if err == nil || !strings.Contains(err.Error(), aclNotFound) {
t.Fatalf("err: %v", err)
}
// A root ACL request should get rejected and not call the server.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
t.Fatalf("should not have called to server")
return nil
}
_, err = agent.resolveToken("deny")
if err == nil || !strings.Contains(err.Error(), rootDenied) {
t.Fatalf("err: %v", err)
}
// The ACL master token should also not call the server, but should give
// us a working agent token.
acl, err := agent.resolveToken("towel")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should be able to read agent")
}
if !acl.AgentWrite(config.NodeName) {
t.Fatalf("should be able to write agent")
}
}
func TestACL_Down_Deny(t *testing.T) {
config := nextConfig()
config.ACLDownPolicy = "deny"
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Resolve with ACLs down.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
return fmt.Errorf("ACLs are broken")
}
acl, err := agent.resolveToken("nope")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if acl.AgentRead(config.NodeName) {
t.Fatalf("should deny")
}
}
func TestACL_Down_Allow(t *testing.T) {
config := nextConfig()
config.ACLDownPolicy = "allow"
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Resolve with ACLs down.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
return fmt.Errorf("ACLs are broken")
}
acl, err := agent.resolveToken("nope")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
}
func TestACL_Down_Extend(t *testing.T) {
config := nextConfig()
config.ACLDownPolicy = "extend-cache"
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Populate the cache for one of the tokens.
m.getPolicyFn = func(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
*reply = structs.ACLPolicy{
Parent: "allow",
Policy: &rawacl.Policy{
Agents: []*rawacl.AgentPolicy{
&rawacl.AgentPolicy{
Node: config.NodeName,
Policy: "read",
},
},
},
}
return nil
}
acl, err := agent.resolveToken("yep")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
if acl.AgentWrite(config.NodeName) {
t.Fatalf("should deny")
}
// Now take down ACLs and make sure a new token fails to resolve.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
return fmt.Errorf("ACLs are broken")
}
acl, err = agent.resolveToken("nope")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if acl.AgentRead(config.NodeName) {
t.Fatalf("should deny")
}
if acl.AgentWrite(config.NodeName) {
t.Fatalf("should deny")
}
// Read the token from the cache while ACLs are broken, which should
// extend.
acl, err = agent.resolveToken("yep")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
if acl.AgentWrite(config.NodeName) {
t.Fatalf("should deny")
}
}
func TestACL_Cache(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Populate the cache for one of the tokens.
m.getPolicyFn = func(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
*reply = structs.ACLPolicy{
ETag: "hash1",
Parent: "deny",
Policy: &rawacl.Policy{
Agents: []*rawacl.AgentPolicy{
&rawacl.AgentPolicy{
Node: config.NodeName,
Policy: "read",
},
},
},
TTL: 10 * time.Millisecond,
}
return nil
}
acl, err := agent.resolveToken("yep")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
if acl.AgentWrite(config.NodeName) {
t.Fatalf("should deny")
}
if acl.NodeRead("nope") {
t.Fatalf("should deny")
}
// Fetch right away and make sure it uses the cache.
m.getPolicyFn = func(*structs.ACLPolicyRequest, *structs.ACLPolicy) error {
t.Fatalf("should not have called to server")
return nil
}
acl, err = agent.resolveToken("yep")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
if acl.AgentWrite(config.NodeName) {
t.Fatalf("should deny")
}
if acl.NodeRead("nope") {
t.Fatalf("should deny")
}
// Wait for the TTL to expire and try again. This time the token will be
// gone.
time.Sleep(20 * time.Millisecond)
m.getPolicyFn = func(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
return errors.New(aclNotFound)
}
_, err = agent.resolveToken("yep")
if err == nil || !strings.Contains(err.Error(), aclNotFound) {
t.Fatalf("err: %v", err)
}
// Page it back in with a new tag and different policy
m.getPolicyFn = func(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
*reply = structs.ACLPolicy{
ETag: "hash2",
Parent: "deny",
Policy: &rawacl.Policy{
Agents: []*rawacl.AgentPolicy{
&rawacl.AgentPolicy{
Node: config.NodeName,
Policy: "write",
},
},
},
TTL: 10 * time.Millisecond,
}
return nil
}
acl, err = agent.resolveToken("yep")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
if !acl.AgentWrite(config.NodeName) {
t.Fatalf("should allow")
}
if acl.NodeRead("nope") {
t.Fatalf("should deny")
}
// Wait for the TTL to expire and try again. This will match the tag
// and not send the policy back, but we should have the old token
// behavior.
time.Sleep(20 * time.Millisecond)
var didRefresh bool
m.getPolicyFn = func(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
*reply = structs.ACLPolicy{
ETag: "hash2",
TTL: 10 * time.Millisecond,
}
didRefresh = true
return nil
}
acl, err = agent.resolveToken("yep")
if err != nil {
t.Fatalf("err: %v", err)
}
if acl == nil {
t.Fatalf("should not be nil")
}
if !acl.AgentRead(config.NodeName) {
t.Fatalf("should allow")
}
if !acl.AgentWrite(config.NodeName) {
t.Fatalf("should allow")
}
if acl.NodeRead("nope") {
t.Fatalf("should deny")
}
if !didRefresh {
t.Fatalf("should refresh")
}
}
// catalogPolicy supplies some standard policies to help with testing the
// catalog-related vet and filter functions.
func catalogPolicy(req *structs.ACLPolicyRequest, reply *structs.ACLPolicy) error {
reply.Policy = &rawacl.Policy{}
switch req.ACL {
case "node-ro":
reply.Policy.Nodes = append(reply.Policy.Nodes,
&rawacl.NodePolicy{Name: "Node", Policy: "read"})
case "node-rw":
reply.Policy.Nodes = append(reply.Policy.Nodes,
&rawacl.NodePolicy{Name: "Node", Policy: "write"})
case "service-ro":
reply.Policy.Services = append(reply.Policy.Services,
&rawacl.ServicePolicy{Name: "service", Policy: "read"})
case "service-rw":
reply.Policy.Services = append(reply.Policy.Services,
&rawacl.ServicePolicy{Name: "service", Policy: "write"})
case "other-rw":
reply.Policy.Services = append(reply.Policy.Services,
&rawacl.ServicePolicy{Name: "other", Policy: "write"})
default:
return fmt.Errorf("unknown token %q", req.ACL)
}
return nil
}
func TestACL_vetServiceRegister(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Register a new service, with permission.
err := agent.vetServiceRegister("service-rw", &structs.NodeService{
ID: "my-service",
Service: "service",
})
if err != nil {
t.Fatalf("err: %v", err)
}
// Register a new service without write privs.
err = agent.vetServiceRegister("service-ro", &structs.NodeService{
ID: "my-service",
Service: "service",
})
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Try to register over a service without write privs to the existing
// service.
agent.state.AddService(&structs.NodeService{
ID: "my-service",
Service: "other",
}, "")
err = agent.vetServiceRegister("service-rw", &structs.NodeService{
ID: "my-service",
Service: "service",
})
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
}
func TestACL_vetServiceUpdate(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Update a service that doesn't exist.
err := agent.vetServiceUpdate("service-rw", "my-service")
if err == nil || !strings.Contains(err.Error(), "Unknown service") {
t.Fatalf("err: %v", err)
}
// Update with write privs.
agent.state.AddService(&structs.NodeService{
ID: "my-service",
Service: "service",
}, "")
err = agent.vetServiceUpdate("service-rw", "my-service")
if err != nil {
t.Fatalf("err: %v", err)
}
// Update without write privs.
err = agent.vetServiceUpdate("service-ro", "my-service")
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
}
func TestACL_vetCheckRegister(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Register a new service check with write privs.
err := agent.vetCheckRegister("service-rw", &structs.HealthCheck{
CheckID: types.CheckID("my-check"),
ServiceID: "my-service",
ServiceName: "service",
})
if err != nil {
t.Fatalf("err: %v", err)
}
// Register a new service check without write privs.
err = agent.vetCheckRegister("service-ro", &structs.HealthCheck{
CheckID: types.CheckID("my-check"),
ServiceID: "my-service",
ServiceName: "service",
})
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Register a new node check with write privs.
err = agent.vetCheckRegister("node-rw", &structs.HealthCheck{
CheckID: types.CheckID("my-check"),
})
if err != nil {
t.Fatalf("err: %v", err)
}
// Register a new node check without write privs.
err = agent.vetCheckRegister("node-ro", &structs.HealthCheck{
CheckID: types.CheckID("my-check"),
})
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Try to register over a service check without write privs to the
// existing service.
agent.state.AddService(&structs.NodeService{
ID: "my-service",
Service: "service",
}, "")
agent.state.AddCheck(&structs.HealthCheck{
CheckID: types.CheckID("my-check"),
ServiceID: "my-service",
ServiceName: "other",
}, "")
err = agent.vetCheckRegister("service-rw", &structs.HealthCheck{
CheckID: types.CheckID("my-check"),
ServiceID: "my-service",
ServiceName: "service",
})
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Try to register over a node check without write privs to the node.
agent.state.AddCheck(&structs.HealthCheck{
CheckID: types.CheckID("my-node-check"),
}, "")
err = agent.vetCheckRegister("service-rw", &structs.HealthCheck{
CheckID: types.CheckID("my-node-check"),
ServiceID: "my-service",
ServiceName: "service",
})
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
}
func TestACL_vetCheckUpdate(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
// Update a check that doesn't exist.
err := agent.vetCheckUpdate("node-rw", "my-check")
if err == nil || !strings.Contains(err.Error(), "Unknown check") {
t.Fatalf("err: %v", err)
}
// Update service check with write privs.
agent.state.AddService(&structs.NodeService{
ID: "my-service",
Service: "service",
}, "")
agent.state.AddCheck(&structs.HealthCheck{
CheckID: types.CheckID("my-service-check"),
ServiceID: "my-service",
ServiceName: "service",
}, "")
err = agent.vetCheckUpdate("service-rw", "my-service-check")
if err != nil {
t.Fatalf("err: %v", err)
}
// Update service check without write privs.
err = agent.vetCheckUpdate("service-ro", "my-service-check")
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Update node check with write privs.
agent.state.AddCheck(&structs.HealthCheck{
CheckID: types.CheckID("my-node-check"),
}, "")
err = agent.vetCheckUpdate("node-rw", "my-node-check")
if err != nil {
t.Fatalf("err: %v", err)
}
// Update without write privs.
err = agent.vetCheckUpdate("node-ro", "my-node-check")
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
}
func TestACL_filterMembers(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
var members []serf.Member
if err := agent.filterMembers("node-ro", &members); err != nil {
t.Fatalf("err: %v", err)
}
if len(members) != 0 {
t.Fatalf("bad: %#v", members)
}
members = []serf.Member{
serf.Member{Name: "Node 1"},
serf.Member{Name: "Nope"},
serf.Member{Name: "Node 2"},
}
if err := agent.filterMembers("node-ro", &members); err != nil {
t.Fatalf("err: %v", err)
}
if len(members) != 2 ||
members[0].Name != "Node 1" ||
members[1].Name != "Node 2" {
t.Fatalf("bad: %#v", members)
}
}
func TestACL_filterServices(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
services := make(map[string]*structs.NodeService)
if err := agent.filterServices("node-ro", &services); err != nil {
t.Fatalf("err: %v", err)
}
services["my-service"] = &structs.NodeService{ID: "my-service", Service: "service"}
services["my-other"] = &structs.NodeService{ID: "my-other", Service: "other"}
if err := agent.filterServices("service-ro", &services); err != nil {
t.Fatalf("err: %v", err)
}
if _, ok := services["my-service"]; !ok {
t.Fatalf("bad: %#v", services)
}
if _, ok := services["my-other"]; ok {
t.Fatalf("bad: %#v", services)
}
}
func TestACL_filterChecks(t *testing.T) {
config := nextConfig()
config.ACLEnforceVersion8 = Bool(true)
dir, agent := makeAgent(t, config)
defer os.RemoveAll(dir)
defer agent.Shutdown()
testutil.WaitForLeader(t, agent.RPC, "dc1")
m := MockServer{catalogPolicy}
if err := agent.InjectEndpoint("ACL", &m); err != nil {
t.Fatalf("err: %v", err)
}
checks := make(map[types.CheckID]*structs.HealthCheck)
if err := agent.filterChecks("node-ro", &checks); err != nil {
t.Fatalf("err: %v", err)
}
checks["my-node"] = &structs.HealthCheck{}
checks["my-service"] = &structs.HealthCheck{ServiceName: "service"}
checks["my-other"] = &structs.HealthCheck{ServiceName: "other"}
if err := agent.filterChecks("service-ro", &checks); err != nil {
t.Fatalf("err: %v", err)
}
if _, ok := checks["my-node"]; ok {
t.Fatalf("bad: %#v", checks)
}
if _, ok := checks["my-service"]; !ok {
t.Fatalf("bad: %#v", checks)
}
if _, ok := checks["my-other"]; ok {
t.Fatalf("bad: %#v", checks)
}
checks["my-node"] = &structs.HealthCheck{}
checks["my-service"] = &structs.HealthCheck{ServiceName: "service"}
checks["my-other"] = &structs.HealthCheck{ServiceName: "other"}
if err := agent.filterChecks("node-ro", &checks); err != nil {
t.Fatalf("err: %v", err)
}
if _, ok := checks["my-node"]; !ok {
t.Fatalf("bad: %#v", checks)
}
if _, ok := checks["my-service"]; ok {
t.Fatalf("bad: %#v", checks)
}
if _, ok := checks["my-other"]; ok {
t.Fatalf("bad: %#v", checks)
}
}

113
command/agent/agent.go
View File

@ -42,11 +42,26 @@ const (
"but no reason was provided. This is a default message."
defaultServiceMaintReason = "Maintenance mode is enabled for this " +
"service, but no reason was provided. This is a default message."
// The meta key prefix reserved for Consul's internal use
metaKeyReservedPrefix = "consul-"
// The maximum number of metadata key pairs allowed to be registered
metaMaxKeyPairs = 64
// The maximum allowed length of a metadata key
metaKeyMaxLength = 128
// The maximum allowed length of a metadata value
metaValueMaxLength = 512
)
var (
// dnsNameRe checks if a name or tag is dns-compatible.
dnsNameRe = regexp.MustCompile(`^[a-zA-Z0-9\-]+$`)
// metaKeyFormat checks if a metadata key string is valid
metaKeyFormat = regexp.MustCompile(`^[a-zA-Z0-9_-]+$`).MatchString
)
/*
@ -74,6 +89,9 @@ type Agent struct {
server *consul.Server
client *consul.Client
// acls is an object that helps manage local ACL enforcement.
acls *aclManager
// state stores a local representation of the node,
// services and checks. Used for anti-entropy.
state localState
@ -211,11 +229,17 @@ func Create(config *Config, logOutput io.Writer, logWriter *logger.LogWriter,
return nil, err
}
// Initialize the ACL manager.
acls, err := newACLManager(config)
if err != nil {
return nil, err
}
agent.acls = acls
// Initialize the local state.
agent.state.Init(config, agent.logger)
// Setup either the client or the server.
var err error
if config.Server {
err = agent.setupServer()
agent.state.SetIface(agent.server)
@ -227,7 +251,8 @@ func Create(config *Config, logOutput io.Writer, logWriter *logger.LogWriter,
Port: agent.config.Ports.Server,
Tags: []string{},
}
agent.state.AddService(&consulService, "")
agent.state.AddService(&consulService, agent.config.GetTokenForAgent())
} else {
err = agent.setupClient()
agent.state.SetIface(agent.client)
@ -236,13 +261,16 @@ func Create(config *Config, logOutput io.Writer, logWriter *logger.LogWriter,
return nil, err
}
// Load checks/services.
// Load checks/services/metadata.
if err := agent.loadServices(config); err != nil {
return nil, err
}
if err := agent.loadChecks(config); err != nil {
return nil, err
}
if err := agent.loadMetadata(config); err != nil {
return nil, err
}
// Start watching for critical services to deregister, based on their
// checks.
@ -363,6 +391,9 @@ func (a *Agent) consulConfig() *consul.Config {
if a.config.ACLToken != "" {
base.ACLToken = a.config.ACLToken
}
if a.config.ACLAgentToken != "" {
base.ACLAgentToken = a.config.ACLAgentToken
}
if a.config.ACLMasterToken != "" {
base.ACLMasterToken = a.config.ACLMasterToken
}
@ -381,6 +412,9 @@ func (a *Agent) consulConfig() *consul.Config {
if a.config.ACLReplicationToken != "" {
base.ACLReplicationToken = a.config.ACLReplicationToken
}
if a.config.ACLEnforceVersion8 != nil {
base.ACLEnforceVersion8 = *a.config.ACLEnforceVersion8
}
if a.config.SessionTTLMinRaw != "" {
base.SessionTTLMin = a.config.SessionTTLMin
}
@ -811,14 +845,11 @@ func (a *Agent) sendCoordinate() {
continue
}
// TODO - Consider adding a distance check so we don't send
// an update if the position hasn't changed by more than a
// threshold.
req := structs.CoordinateUpdateRequest{
Datacenter: a.config.Datacenter,
Node: a.config.NodeName,
Coord: c,
WriteRequest: structs.WriteRequest{Token: a.config.ACLToken},
WriteRequest: structs.WriteRequest{Token: a.config.GetTokenForAgent()},
}
var reply struct{}
if err := a.RPC("Coordinate.Update", &req, &reply); err != nil {
@ -1664,6 +1695,74 @@ func (a *Agent) restoreCheckState(snap map[types.CheckID]*structs.HealthCheck) {
}
}
// loadMetadata loads node metadata fields from the agent config and
// updates them on the local agent.
func (a *Agent) loadMetadata(conf *Config) error {
a.state.Lock()
defer a.state.Unlock()
for key, value := range conf.Meta {
a.state.metadata[key] = value
}
a.state.changeMade()
return nil
}
// parseMetaPair parses a key/value pair of the form key:value
func parseMetaPair(raw string) (string, string) {
pair := strings.SplitN(raw, ":", 2)
if len(pair) == 2 {
return pair[0], pair[1]
} else {
return pair[0], ""
}
}
// validateMeta validates a set of key/value pairs from the agent config
func validateMetadata(meta map[string]string) error {
if len(meta) > metaMaxKeyPairs {
return fmt.Errorf("Node metadata cannot contain more than %d key/value pairs", metaMaxKeyPairs)
}
for key, value := range meta {
if err := validateMetaPair(key, value); err != nil {
return fmt.Errorf("Couldn't load metadata pair ('%s', '%s'): %s", key, value, err)
}
}
return nil
}
// validateMetaPair checks that the given key/value pair is in a valid format
func validateMetaPair(key, value string) error {
if key == "" {
return fmt.Errorf("Key cannot be blank")
}
if !metaKeyFormat(key) {
return fmt.Errorf("Key contains invalid characters")
}
if len(key) > metaKeyMaxLength {
return fmt.Errorf("Key is too long (limit: %d characters)", metaKeyMaxLength)
}
if strings.HasPrefix(key, metaKeyReservedPrefix) {
return fmt.Errorf("Key prefix '%s' is reserved for internal use", metaKeyReservedPrefix)
}
if len(value) > metaValueMaxLength {
return fmt.Errorf("Value is too long (limit: %d characters)", metaValueMaxLength)
}
return nil
}
// unloadMetadata resets the local metadata state
func (a *Agent) unloadMetadata() {
a.state.Lock()
defer a.state.Unlock()
a.state.metadata = make(map[string]string)
}
// serviceMaintCheckID returns the ID of a given service's maintenance check
func serviceMaintCheckID(serviceID string) types.CheckID {
return types.CheckID(structs.ServiceMaintPrefix + serviceID)

224
command/agent/agent_endpoint.go
View File

@ -20,6 +20,7 @@ type AgentSelf struct {
Coord *coordinate.Coordinate
Member serf.Member
Stats map[string]map[string]string
Meta map[string]string
}
func (s *HTTPServer) AgentSelf(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
@ -31,11 +32,23 @@ func (s *HTTPServer) AgentSelf(resp http.ResponseWriter, req *http.Request) (int
}
}
// Fetch the ACL token, if any, and enforce agent policy.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.AgentRead(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
return AgentSelf{
Config: s.agent.config,
Coord: c,
Member: s.agent.LocalMember(),
Stats: s.agent.Stats(),
Meta: s.agent.state.Metadata(),
}, nil
}
@ -45,9 +58,19 @@ func (s *HTTPServer) AgentReload(resp http.ResponseWriter, req *http.Request) (i
return nil, nil
}
errCh := make(chan error, 0)
// Fetch the ACL token, if any, and enforce agent policy.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.AgentWrite(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
// Trigger the reload
errCh := make(chan error, 0)
select {
case <-s.agent.ShutdownCh():
return nil, fmt.Errorf("Agent was shutdown before reload could be completed")
@ -64,29 +87,64 @@ func (s *HTTPServer) AgentReload(resp http.ResponseWriter, req *http.Request) (i
}
func (s *HTTPServer) AgentServices(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Fetch the ACL token, if any.
var token string
s.parseToken(req, &token)
services := s.agent.state.Services()
if err := s.agent.filterServices(token, &services); err != nil {
return nil, err
}
return services, nil
}
func (s *HTTPServer) AgentChecks(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Fetch the ACL token, if any.
var token string
s.parseToken(req, &token)
checks := s.agent.state.Checks()
if err := s.agent.filterChecks(token, &checks); err != nil {
return nil, err
}
return checks, nil
}
func (s *HTTPServer) AgentMembers(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Fetch the ACL token, if any.
var token string
s.parseToken(req, &token)
// Check if the WAN is being queried
wan := false
if other := req.URL.Query().Get("wan"); other != "" {
wan = true
}
var members []serf.Member
if wan {
return s.agent.WANMembers(), nil
members = s.agent.WANMembers()
} else {
return s.agent.LANMembers(), nil
members = s.agent.LANMembers()
}
if err := s.agent.filterMembers(token, &members); err != nil {
return nil, err
}
return members, nil
}
func (s *HTTPServer) AgentJoin(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Fetch the ACL token, if any, and enforce agent policy.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.AgentWrite(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
// Check if the WAN is being queried
wan := false
if other := req.URL.Query().Get("wan"); other != "" {
@ -110,6 +168,17 @@ func (s *HTTPServer) AgentLeave(resp http.ResponseWriter, req *http.Request) (in
return nil, nil
}
// Fetch the ACL token, if any, and enforce agent policy.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.AgentWrite(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
if err := s.agent.Leave(); err != nil {
return nil, err
}
@ -117,15 +186,35 @@ func (s *HTTPServer) AgentLeave(resp http.ResponseWriter, req *http.Request) (in
}
func (s *HTTPServer) AgentForceLeave(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Fetch the ACL token, if any, and enforce agent policy.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.AgentWrite(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
addr := strings.TrimPrefix(req.URL.Path, "/v1/agent/force-leave/")
return nil, s.agent.ForceLeave(addr)
}
// syncChanges is a helper function which wraps a blocking call to sync
// services and checks to the server. If the operation fails, we only
// only warn because the write did succeed and anti-entropy will sync later.
func (s *HTTPServer) syncChanges() {
if err := s.agent.state.syncChanges(); err != nil {
s.logger.Printf("[ERR] agent: failed to sync changes: %v", err)
}
}
const invalidCheckMessage = "Must provide TTL or Script/DockerContainerID/HTTP/TCP and Interval"
func (s *HTTPServer) AgentRegisterCheck(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
var args CheckDefinition
// Fixup the type decode of TTL or Interval
// Fixup the type decode of TTL or Interval.
decodeCB := func(raw interface{}) error {
return FixupCheckType(raw)
}
@ -135,7 +224,7 @@ func (s *HTTPServer) AgentRegisterCheck(resp http.ResponseWriter, req *http.Requ
return nil, nil
}
// Verify the check has a name
// Verify the check has a name.
if args.Name == "" {
resp.WriteHeader(400)
resp.Write([]byte("Missing check name"))
@ -148,10 +237,10 @@ func (s *HTTPServer) AgentRegisterCheck(resp http.ResponseWriter, req *http.Requ
return nil, nil
}
// Construct the health check
// Construct the health check.
health := args.HealthCheck(s.agent.config.NodeName)
// Verify the check type
// Verify the check type.
chkType := &args.CheckType
if !chkType.Valid() {
resp.WriteHeader(400)
@ -159,11 +248,14 @@ func (s *HTTPServer) AgentRegisterCheck(resp http.ResponseWriter, req *http.Requ
return nil, nil
}
// Get the provided token, if any
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetCheckRegister(token, health); err != nil {
return nil, err
}
// Add the check
// Add the check.
if err := s.agent.AddCheck(health, chkType, true, token); err != nil {
return nil, err
}
@ -173,6 +265,14 @@ func (s *HTTPServer) AgentRegisterCheck(resp http.ResponseWriter, req *http.Requ
func (s *HTTPServer) AgentDeregisterCheck(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
checkID := types.CheckID(strings.TrimPrefix(req.URL.Path, "/v1/agent/check/deregister/"))
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetCheckUpdate(token, checkID); err != nil {
return nil, err
}
if err := s.agent.RemoveCheck(checkID, true); err != nil {
return nil, err
}
@ -183,6 +283,14 @@ func (s *HTTPServer) AgentDeregisterCheck(resp http.ResponseWriter, req *http.Re
func (s *HTTPServer) AgentCheckPass(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
checkID := types.CheckID(strings.TrimPrefix(req.URL.Path, "/v1/agent/check/pass/"))
note := req.URL.Query().Get("note")
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetCheckUpdate(token, checkID); err != nil {
return nil, err
}
if err := s.agent.updateTTLCheck(checkID, structs.HealthPassing, note); err != nil {
return nil, err
}
@ -193,6 +301,14 @@ func (s *HTTPServer) AgentCheckPass(resp http.ResponseWriter, req *http.Request)
func (s *HTTPServer) AgentCheckWarn(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
checkID := types.CheckID(strings.TrimPrefix(req.URL.Path, "/v1/agent/check/warn/"))
note := req.URL.Query().Get("note")
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetCheckUpdate(token, checkID); err != nil {
return nil, err
}
if err := s.agent.updateTTLCheck(checkID, structs.HealthWarning, note); err != nil {
return nil, err
}
@ -203,6 +319,14 @@ func (s *HTTPServer) AgentCheckWarn(resp http.ResponseWriter, req *http.Request)
func (s *HTTPServer) AgentCheckFail(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
checkID := types.CheckID(strings.TrimPrefix(req.URL.Path, "/v1/agent/check/fail/"))
note := req.URL.Query().Get("note")
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetCheckUpdate(token, checkID); err != nil {
return nil, err
}
if err := s.agent.updateTTLCheck(checkID, structs.HealthCritical, note); err != nil {
return nil, err
}
@ -255,6 +379,14 @@ func (s *HTTPServer) AgentCheckUpdate(resp http.ResponseWriter, req *http.Reques
}
checkID := types.CheckID(strings.TrimPrefix(req.URL.Path, "/v1/agent/check/update/"))
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetCheckUpdate(token, checkID); err != nil {
return nil, err
}
if err := s.agent.updateTTLCheck(checkID, update.Status, update.Output); err != nil {
return nil, err
}
@ -264,7 +396,7 @@ func (s *HTTPServer) AgentCheckUpdate(resp http.ResponseWriter, req *http.Reques
func (s *HTTPServer) AgentRegisterService(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
var args ServiceDefinition
// Fixup the type decode of TTL or Interval if a check if provided
// Fixup the type decode of TTL or Interval if a check if provided.
decodeCB := func(raw interface{}) error {
rawMap, ok := raw.(map[string]interface{})
if !ok {
@ -297,17 +429,17 @@ func (s *HTTPServer) AgentRegisterService(resp http.ResponseWriter, req *http.Re
return nil, nil
}
// Verify the service has a name
// Verify the service has a name.
if args.Name == "" {
resp.WriteHeader(400)
resp.Write([]byte("Missing service name"))
return nil, nil
}
// Get the node service
// Get the node service.
ns := args.NodeService()
// Verify the check type
// Verify the check type.
chkTypes := args.CheckTypes()
for _, check := range chkTypes {
if check.Status != "" && !structs.ValidStatus(check.Status) {
@ -322,11 +454,14 @@ func (s *HTTPServer) AgentRegisterService(resp http.ResponseWriter, req *http.Re
}
}
// Get the provided token, if any
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetServiceRegister(token, ns); err != nil {
return nil, err
}
// Add the check
// Add the service.
if err := s.agent.AddService(ns, chkTypes, true, token); err != nil {
return nil, err
}
@ -336,6 +471,14 @@ func (s *HTTPServer) AgentRegisterService(resp http.ResponseWriter, req *http.Re
func (s *HTTPServer) AgentDeregisterService(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
serviceID := strings.TrimPrefix(req.URL.Path, "/v1/agent/service/deregister/")
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetServiceUpdate(token, serviceID); err != nil {
return nil, err
}
if err := s.agent.RemoveService(serviceID, true); err != nil {
return nil, err
}
@ -374,9 +517,12 @@ func (s *HTTPServer) AgentServiceMaintenance(resp http.ResponseWriter, req *http
return nil, nil
}
// Get the provided token, if any
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
if err := s.agent.vetServiceUpdate(token, serviceID); err != nil {
return nil, err
}
if enable {
reason := params.Get("reason")
@ -419,9 +565,16 @@ func (s *HTTPServer) AgentNodeMaintenance(resp http.ResponseWriter, req *http.Re
return nil, nil
}
// Get the provided token, if any
// Get the provided token, if any, and vet against any ACL policies.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.NodeWrite(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
if enable {
s.agent.EnableNodeMaintenance(params.Get("reason"), token)
@ -433,31 +586,33 @@ func (s *HTTPServer) AgentNodeMaintenance(resp http.ResponseWriter, req *http.Re
}
func (s *HTTPServer) AgentMonitor(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Only GET supported
// Only GET supported.
if req.Method != "GET" {
resp.WriteHeader(405)
return nil, nil
}
var args structs.DCSpecificRequest
args.Datacenter = s.agent.config.Datacenter
s.parseToken(req, &args.Token)
// Validate that the given token has operator permissions
var reply structs.RaftConfigurationResponse
if err := s.agent.RPC("Operator.RaftGetConfiguration", &args, &reply); err != nil {
// Fetch the ACL token, if any, and enforce agent policy.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
if acl != nil && !acl.AgentRead(s.agent.config.NodeName) {
return nil, permissionDeniedErr
}
// Get the provided loglevel
// Get the provided loglevel.
logLevel := req.URL.Query().Get("loglevel")
if logLevel == "" {
logLevel = "INFO"
}
// Upper case the log level
// Upper case the level since that's required by the filter.
logLevel = strings.ToUpper(logLevel)
// Create a level filter
// Create a level filter and flusher.
filter := logger.LevelFilter()
filter.MinLevel = logutils.LogLevel(logLevel)
if !logger.ValidateLevelFilter(filter.MinLevel, filter) {
@ -465,13 +620,12 @@ func (s *HTTPServer) AgentMonitor(resp http.ResponseWriter, req *http.Request) (
resp.Write([]byte(fmt.Sprintf("Unknown log level: %s", filter.MinLevel)))
return nil, nil
}
flusher, ok := resp.(http.Flusher)
if !ok {
return nil, fmt.Errorf("Streaming not supported")
}
// Set up a log handler
// Set up a log handler.
handler := &httpLogHandler{
filter: filter,
logCh: make(chan string, 512),
@ -479,10 +633,9 @@ func (s *HTTPServer) AgentMonitor(resp http.ResponseWriter, req *http.Request) (
}
s.agent.logWriter.RegisterHandler(handler)
defer s.agent.logWriter.DeregisterHandler(handler)
notify := resp.(http.CloseNotifier).CloseNotify()
// Stream logs until the connection is closed
// Stream logs until the connection is closed.
for {
select {
case <-notify:
@ -500,15 +653,6 @@ func (s *HTTPServer) AgentMonitor(resp http.ResponseWriter, req *http.Request) (
return nil, nil
}
// syncChanges is a helper function which wraps a blocking call to sync
// services and checks to the server. If the operation fails, we only
// only warn because the write did succeed and anti-entropy will sync later.
func (s *HTTPServer) syncChanges() {
if err := s.agent.state.syncChanges(); err != nil {
s.logger.Printf("[ERR] agent: failed to sync changes: %v", err)
}
}
type httpLogHandler struct {
filter *logutils.LevelFilter
logCh chan string

929
command/agent/agent_endpoint_test.go
View File

File diff suppressed because it is too large Load Diff

65
command/agent/agent_test.go
View File

@ -19,6 +19,7 @@ import (
"github.com/hashicorp/consul/logger"
"github.com/hashicorp/consul/testutil"
"github.com/hashicorp/raft"
"strings"
)
const (
@ -55,6 +56,7 @@ func nextConfig() *Config {
conf.Ports.SerfWan = basePortNumber + idx + portOffsetSerfWan
conf.Ports.Server = basePortNumber + idx + portOffsetServer
conf.Server = true
conf.ACLEnforceVersion8 = Bool(false)
conf.ACLDatacenter = "dc1"
conf.ACLMasterToken = "root"
@ -1850,6 +1852,69 @@ func TestAgent_purgeCheckState(t *testing.T) {
}
}
func TestAgent_metadata(t *testing.T) {
// Load a valid set of key/value pairs
meta := map[string]string{
"key1": "value1",
"key2": "value2",
}
// Should succeed
if err := validateMetadata(meta); err != nil {
t.Fatalf("err: %s", err)
}
// Should get error
meta = map[string]string{
"": "value1",
}
if err := validateMetadata(meta); !strings.Contains(err.Error(), "Couldn't load metadata pair") {
t.Fatalf("should have failed")
}
// Should get error
meta = make(map[string]string)
for i := 0; i < metaMaxKeyPairs+1; i++ {
meta[string(i)] = "value"
}
if err := validateMetadata(meta); !strings.Contains(err.Error(), "cannot contain more than") {
t.Fatalf("should have failed")
}
}
func TestAgent_validateMetaPair(t *testing.T) {
longKey := strings.Repeat("a", metaKeyMaxLength+1)
longValue := strings.Repeat("b", metaValueMaxLength+1)
pairs := []struct {
Key string
Value string
Error string
}{
// valid pair
{"key", "value", ""},
// invalid, blank key
{"", "value", "cannot be blank"},
// allowed special chars in key name
{"k_e-y", "value", ""},
// disallowed special chars in key name
{"(%key&)", "value", "invalid characters"},
// key too long
{longKey, "value", "Key is too long"},
// reserved prefix
{metaKeyReservedPrefix + "key", "value", "reserved for internal use"},
// value too long
{"key", longValue, "Value is too long"},
}
for _, pair := range pairs {
err := validateMetaPair(pair.Key, pair.Value)
if pair.Error == "" && err != nil {
t.Fatalf("should have succeeded: %v, %v", pair, err)
} else if pair.Error != "" && !strings.Contains(err.Error(), pair.Error) {
t.Fatalf("should have failed: %v, %v", pair, err)
}
}
}
func TestAgent_GetCoordinate(t *testing.T) {
check := func(server bool) {
config := nextConfig()

20
command/agent/bindata_assetfs.go
View File

File diff suppressed because one or more lines are too long

2
command/agent/catalog_endpoint.go
View File

@ -64,6 +64,7 @@ func (s *HTTPServer) CatalogNodes(resp http.ResponseWriter, req *http.Request) (
// Setup the request
args := structs.DCSpecificRequest{}
s.parseSource(req, &args.Source)
args.NodeMetaFilters = s.parseMetaFilter(req)
if done := s.parse(resp, req, &args.Datacenter, &args.QueryOptions); done {
return nil, nil
}
@ -85,6 +86,7 @@ func (s *HTTPServer) CatalogNodes(resp http.ResponseWriter, req *http.Request) (
func (s *HTTPServer) CatalogServices(resp http.ResponseWriter, req *http.Request) (interface{}, error) {
// Set default DC
args := structs.DCSpecificRequest{}
args.NodeMetaFilters = s.parseMetaFilter(req)
if done := s.parse(resp, req, &args.Datacenter, &args.QueryOptions); done {
return nil, nil
}

95
command/agent/catalog_endpoint_test.go
View File

@ -145,6 +145,53 @@ func TestCatalogNodes(t *testing.T) {
}
}
func TestCatalogNodes_MetaFilter(t *testing.T) {
dir, srv := makeHTTPServer(t)
defer os.RemoveAll(dir)
defer srv.Shutdown()
defer srv.agent.Shutdown()
testutil.WaitForLeader(t, srv.agent.RPC, "dc1")
// Register a node with a meta field
args := &structs.RegisterRequest{
Datacenter: "dc1",
Node: "foo",
Address: "127.0.0.1",
NodeMeta: map[string]string{
"somekey": "somevalue",
},
}
var out struct{}
if err := srv.agent.RPC("Catalog.Register", args, &out); err != nil {
t.Fatalf("err: %v", err)
}
req, err := http.NewRequest("GET", "/v1/catalog/nodes?node-meta=somekey:somevalue", nil)
if err != nil {
t.Fatalf("err: %v", err)
}
resp := httptest.NewRecorder()
obj, err := srv.CatalogNodes(resp, req)
if err != nil {
t.Fatalf("err: %v", err)
}
// Verify an index is set
assertIndex(t, resp)
// Verify we only get the node with the correct meta field back
nodes := obj.(structs.Nodes)
if len(nodes) != 1 {
t.Fatalf("bad: %v", obj)
}
if v, ok := nodes[0].Meta["somekey"]; !ok || v != "somevalue" {
t.Fatalf("bad: %v", nodes[0].Meta)
}
}
func TestCatalogNodes_WanTranslation(t *testing.T) {
dir1, srv1 := makeHTTPServerWithConfig(t,
func(c *Config) {
@ -449,6 +496,54 @@ func TestCatalogServices(t *testing.T) {
}
}
func TestCatalogServices_NodeMetaFilter(t *testing.T) {
dir, srv := makeHTTPServer(t)
defer os.RemoveAll(dir)
defer srv.Shutdown()
defer srv.agent.Shutdown()
testutil.WaitForLeader(t, srv.agent.RPC, "dc1")
// Register node
args := &structs.RegisterRequest{
Datacenter: "dc1",
Node: "foo",
Address: "127.0.0.1",
NodeMeta: map[string]string{
"somekey": "somevalue",
},
Service: &structs.NodeService{
Service: "api",
},
}
var out struct{}
if err := srv.agent.RPC("Catalog.Register", args, &out); err != nil {
t.Fatalf("err: %v", err)
}
req, err := http.NewRequest("GET", "/v1/catalog/services?node-meta=somekey:somevalue", nil)
if err != nil {
t.Fatalf("err: %v", err)
}
resp := httptest.NewRecorder()
obj, err := srv.CatalogServices(resp, req)
if err != nil {
t.Fatalf("err: %v", err)
}
assertIndex(t, resp)
services := obj.(structs.Services)
if len(services) != 1 {
t.Fatalf("bad: %v", obj)
}
if _, ok := services[args.Service.Service]; !ok {
t.Fatalf("bad: %v", services)
}
}
func TestCatalogServiceNodes(t *testing.T) {
dir, srv := makeHTTPServer(t)
defer os.RemoveAll(dir)

60
command/agent/command.go
View File

@ -80,12 +80,14 @@ func (c *Command) readConfig() *Config {
var dnsRecursors []string
var dev bool
var dcDeprecated string
var nodeMeta []string
cmdFlags := flag.NewFlagSet("agent", flag.ContinueOnError)
cmdFlags.Usage = func() { c.Ui.Output(c.Help()) }
cmdFlags.Var((*AppendSliceValue)(&configFiles), "config-file", "json file to read config from")
cmdFlags.Var((*AppendSliceValue)(&configFiles), "config-dir", "directory of json files to read")
cmdFlags.Var((*AppendSliceValue)(&dnsRecursors), "recursor", "address of an upstream DNS server")
cmdFlags.Var((*AppendSliceValue)(&nodeMeta), "node-meta", "arbitrary metadata key/value pair")
cmdFlags.BoolVar(&dev, "dev", false, "development server mode")
cmdFlags.StringVar(&cmdConfig.LogLevel, "log-level", "", "log level")
@ -176,6 +178,14 @@ func (c *Command) readConfig() *Config {
cmdConfig.RetryIntervalWan = dur
}
if len(nodeMeta) > 0 {
cmdConfig.Meta = make(map[string]string)
for _, entry := range nodeMeta {
key, value := parseMetaPair(entry)
cmdConfig.Meta[key] = value
}
}
var config *Config
if dev {
config = DevConfig()
@ -220,10 +230,22 @@ func (c *Command) readConfig() *Config {
config.SkipLeaveOnInt = Bool(config.Server)
}
// Ensure we have a data directory
if config.DataDir == "" && !dev {
c.Ui.Error("Must specify data directory using -data-dir")
return nil
// Ensure we have a data directory if we are not in dev mode.
if !dev {
if config.DataDir == "" {
c.Ui.Error("Must specify data directory using -data-dir")
return nil
}
if finfo, err := os.Stat(config.DataDir); err != nil {
if !os.IsNotExist(err) {
c.Ui.Error(fmt.Sprintf("Error getting data-dir: %s", err))
return nil
}
} else if !finfo.IsDir() {
c.Ui.Error(fmt.Sprintf("The data-dir specified at %q is not a directory", config.DataDir))
return nil
}
}
// Ensure all endpoints are unique
@ -373,6 +395,12 @@ func (c *Command) readConfig() *Config {
return nil
}
// Verify the node metadata entries are valid
if err := validateMetadata(config.Meta); err != nil {
c.Ui.Error(fmt.Sprintf("Failed to parse node metadata: %v", err))
return nil
}
// Set the version info
config.Revision = c.Revision
config.Version = c.Version
@ -812,7 +840,7 @@ func (c *Command) retryJoin(config *Config, errCh chan<- struct{}) {
case ec2Enabled:
servers, err = config.discoverEc2Hosts(logger)
if err != nil {
logger.Printf("[ERROR] agent: Unable to query EC2 insances: %s", err)
logger.Printf("[ERROR] agent: Unable to query EC2 instances: %s", err)
}
logger.Printf("[INFO] agent: Discovered %d servers from EC2...", len(servers))
case config.RetryJoinGCE.TagValue != "":
@ -1119,6 +1147,7 @@ func (c *Command) Run(args []string) int {
func (c *Command) handleSignals(config *Config, retryJoin <-chan struct{}, retryJoinWan <-chan struct{}) int {
signalCh := make(chan os.Signal, 4)
signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM, syscall.SIGHUP)
signal.Notify(signalCh, os.Interrupt, syscall.SIGTERM, syscall.SIGHUP, syscall.SIGPIPE)
// Wait for a signal
WAIT:
@ -1144,6 +1173,11 @@ WAIT:
}
c.Ui.Output(fmt.Sprintf("Caught signal: %v", sig))
// Skip SIGPIPE signals
if sig == syscall.SIGPIPE {
goto WAIT
}
// Check if this is a SIGHUP
if sig == syscall.SIGHUP {
conf, err := c.handleReload(config)
@ -1226,7 +1260,7 @@ func (c *Command) handleReload(config *Config) (*Config, error) {
snap := c.agent.snapshotCheckState()
defer c.agent.restoreCheckState(snap)
// First unload all checks and services. This lets us begin the reload
// First unload all checks, services, and metadata. This lets us begin the reload
// with a clean slate.
if err := c.agent.unloadServices(); err != nil {
errs = multierror.Append(errs, fmt.Errorf("Failed unloading services: %s", err))
@ -1236,8 +1270,9 @@ func (c *Command) handleReload(config *Config) (*Config, error) {
errs = multierror.Append(errs, fmt.Errorf("Failed unloading checks: %s", err))
return nil, errs
}
c.agent.unloadMetadata()
// Reload services and check definitions.
// Reload service/check definitions and metadata.
if err := c.agent.loadServices(newConf); err != nil {
errs = multierror.Append(errs, fmt.Errorf("Failed reloading services: %s", err))
return nil, errs
@ -1246,6 +1281,10 @@ func (c *Command) handleReload(config *Config) (*Config, error) {
errs = multierror.Append(errs, fmt.Errorf("Failed reloading checks: %s", err))
return nil, errs
}
if err := c.agent.loadMetadata(newConf); err != nil {
errs = multierror.Append(errs, fmt.Errorf("Failed reloading metadata: %s", err))
return nil, errs
}
// Get the new client listener addr
httpAddr, err := newConf.ClientListener(config.Addresses.HTTP, config.Ports.HTTP)
@ -1298,6 +1337,10 @@ func (c *Command) setupScadaConn(config *Config) error {
return nil
}
c.Ui.Error("WARNING: The hosted version of Consul Enterprise will be deprecated " +
"on March 7th, 2017. For details, see " +
"https://atlas.hashicorp.com/help/consul/alternatives")
scadaConfig := &scada.Config{
Service: "consul",
Version: fmt.Sprintf("%s%s", config.Version, config.VersionPrerelease),
@ -1405,6 +1448,9 @@ Options:
-log-level=info Log level of the agent.
-node=hostname Name of this node. Must be unique in the
cluster
-node-meta=key:value An arbitrary metadata key/value pair for
this node.
This can be specified multiple times.
-protocol=N Sets the protocol version. Defaults to
latest.
-rejoin Ignores a previous leave and attempts to

26
command/agent/command_test.go
View File

@ -13,6 +13,7 @@ import (
"github.com/hashicorp/consul/logger"
"github.com/hashicorp/consul/testutil"
"github.com/mitchellh/cli"
"reflect"
)
func TestCommand_implements(t *testing.T) {
@ -129,6 +130,7 @@ func TestReadCliConfig(t *testing.T) {
"-advertise-wan", "1.2.3.4",
"-serf-wan-bind", "4.3.2.1",
"-serf-lan-bind", "4.3.2.2",
"-node-meta", "somekey:somevalue",
},
ShutdownCh: shutdownCh,
Ui: new(cli.MockUi),
@ -144,6 +146,30 @@ func TestReadCliConfig(t *testing.T) {
if config.SerfLanBindAddr != "4.3.2.2" {
t.Fatalf("expected -serf-lan-bind 4.3.2.2 got %s", config.SerfLanBindAddr)
}
if len(config.Meta) != 1 || config.Meta["somekey"] != "somevalue" {
t.Fatalf("expected somekey=somevalue, got %v", config.Meta)
}
}
// Test multiple node meta flags
{
cmd := &Command{
args: []string{
"-data-dir", tmpDir,
"-node-meta", "somekey:somevalue",
"-node-meta", "otherkey:othervalue",
},
ShutdownCh: shutdownCh,
Ui: new(cli.MockUi),
}
expected := map[string]string{
"somekey": "somevalue",
"otherkey": "othervalue",
}
config := cmd.readConfig()
if !reflect.DeepEqual(config.Meta, expected) {
t.Fatalf("bad: %v %v", config.Meta, expected)
}
}
// Test LeaveOnTerm and SkipLeaveOnInt defaults for server mode

67
command/agent/config.go
View File

@ -367,6 +367,11 @@ type Config struct {
// they are configured with TranslateWanAddrs set to true.
TaggedAddresses map[string]string
// Node metadata key/value pairs. These are excluded from JSON output
// because they can be reloaded and might be stale when shown from the
// config instead of the local state.
Meta map[string]string `mapstructure:"node_meta" json:"-"`
// LeaveOnTerm controls if Serf does a graceful leave when receiving
// the TERM signal. Defaults true on clients, false on servers. This can
// be changed on reload.
@ -516,6 +521,16 @@ type Config struct {
// token is not provided. If not configured the 'anonymous' token is used.
ACLToken string `mapstructure:"acl_token" json:"-"`
// ACLAgentMasterToken is a special token that has full read and write
// privileges for this agent, and can be used to call agent endpoints
// when no servers are available.
ACLAgentMasterToken string `mapstructure:"acl_agent_master_token" json:"-"`
// ACLAgentToken is the default token used to make requests for the agent
// itself, such as for registering itself with the catalog. If not
// configured, the 'acl_token' will be used.
ACLAgentToken string `mapstructure:"acl_agent_token" json:"-"`
// ACLMasterToken is used to bootstrap the ACL system. It should be specified
// on the servers in the ACLDatacenter. When the leader comes online, it ensures
// that the Master token is available. This provides the initial token.
@ -537,9 +552,15 @@ type Config struct {
// white-lists.
ACLDefaultPolicy string `mapstructure:"acl_default_policy"`
// ACLDisabledTTL is used by clients to determine how long they will
// wait to check again with the servers if they discover ACLs are not
// enabled.
ACLDisabledTTL time.Duration `mapstructure:"-"`
// ACLDownPolicy is used to control the ACL interaction when we cannot
// reach the ACLDatacenter and the token is not in the cache.
// There are two modes:
// * allow - Allow all requests
// * deny - Deny all requests
// * extend-cache - Ignore the cache expiration, and allow cached
// ACL's to be used to service requests. This
@ -553,6 +574,10 @@ type Config struct {
// other than the ACLDatacenter.
ACLReplicationToken string `mapstructure:"acl_replication_token" json:"-"`
// ACLEnforceVersion8 is used to gate a set of ACL policy features that
// are opt-in prior to Consul 0.8 and opt-out in Consul 0.8 and later.
ACLEnforceVersion8 *bool `mapstructure:"acl_enforce_version_8"`
// Watches are used to monitor various endpoints and to invoke a
// handler to act appropriately. These are managed entirely in the
// agent layer using the standard APIs.
@ -718,6 +743,7 @@ func DefaultConfig() *Config {
Telemetry: Telemetry{
StatsitePrefix: "consul",
},
Meta: make(map[string]string),
SyslogFacility: "LOCAL0",
Protocol: consul.ProtocolVersion2Compatible,
CheckUpdateInterval: 5 * time.Minute,
@ -733,11 +759,13 @@ func DefaultConfig() *Config {
SyncCoordinateRateTarget: 64.0, // updates / second
SyncCoordinateIntervalMin: 15 * time.Second,
ACLTTL: 30 * time.Second,
ACLDownPolicy: "extend-cache",
ACLDefaultPolicy: "allow",
RetryInterval: 30 * time.Second,
RetryIntervalWan: 30 * time.Second,
ACLTTL: 30 * time.Second,
ACLDownPolicy: "extend-cache",
ACLDefaultPolicy: "allow",
ACLDisabledTTL: 120 * time.Second,
ACLEnforceVersion8: Bool(false),
RetryInterval: 30 * time.Second,
RetryIntervalWan: 30 * time.Second,
}
}
@ -779,6 +807,18 @@ func (c *Config) ClientListener(override string, port int) (net.Addr, error) {
return &net.TCPAddr{IP: ip, Port: port}, nil
}
// GetTokenForAgent returns the token the agent should use for its own internal
// operations, such as registering itself with the catalog.
func (c *Config) GetTokenForAgent() string {
if c.ACLAgentToken != "" {
return c.ACLAgentToken
} else if c.ACLToken != "" {
return c.ACLToken
} else {
return ""
}
}
// DecodeConfig reads the configuration from the given reader in JSON
// format and decodes it into a proper Config structure.
func DecodeConfig(r io.Reader) (*Config, error) {
@ -1501,6 +1541,12 @@ func MergeConfig(a, b *Config) *Config {
if b.ACLToken != "" {
result.ACLToken = b.ACLToken
}
if b.ACLAgentMasterToken != "" {
result.ACLAgentMasterToken = b.ACLAgentMasterToken
}
if b.ACLAgentToken != "" {
result.ACLAgentToken = b.ACLAgentToken
}
if b.ACLMasterToken != "" {
result.ACLMasterToken = b.ACLMasterToken
}
@ -1520,6 +1566,9 @@ func MergeConfig(a, b *Config) *Config {
if b.ACLReplicationToken != "" {
result.ACLReplicationToken = b.ACLReplicationToken
}
if b.ACLEnforceVersion8 != nil {
result.ACLEnforceVersion8 = b.ACLEnforceVersion8
}
if len(b.Watches) != 0 {
result.Watches = append(result.Watches, b.Watches...)
}
@ -1574,6 +1623,14 @@ func MergeConfig(a, b *Config) *Config {
result.HTTPAPIResponseHeaders[field] = value
}
}
if len(b.Meta) != 0 {
if result.Meta == nil {
result.Meta = make(map[string]string)
}
for field, value := range b.Meta {
result.Meta[field] = value
}
}
// Copy the start join addresses
result.StartJoin = make([]string, 0, len(a.StartJoin)+len(b.StartJoin))

79
command/agent/config_test.go
View File

@ -281,6 +281,19 @@ func TestDecodeConfig(t *testing.T) {
t.Fatalf("bad: %#v", config)
}
// Node metadata fields
input = `{"node_meta": {"thing1": "1", "thing2": "2"}}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
if err != nil {
t.Fatalf("err: %s", err)
}
if v, ok := config.Meta["thing1"]; !ok || v != "1" {
t.Fatalf("bad: %#v", config)
}
if v, ok := config.Meta["thing2"]; !ok || v != "2" {
t.Fatalf("bad: %#v", config)
}
// leave_on_terminate
input = `{"leave_on_terminate": true}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
@ -643,7 +656,8 @@ func TestDecodeConfig(t *testing.T) {
}
// ACLs
input = `{"acl_token": "1234", "acl_datacenter": "dc2",
input = `{"acl_token": "1111", "acl_agent_master_token": "2222",
"acl_agent_token": "3333", "acl_datacenter": "dc2",
"acl_ttl": "60s", "acl_down_policy": "deny",
"acl_default_policy": "deny", "acl_master_token": "2345",
"acl_replication_token": "8675309"}`
@ -652,7 +666,13 @@ func TestDecodeConfig(t *testing.T) {
t.Fatalf("err: %s", err)
}
if config.ACLToken != "1234" {
if config.ACLToken != "1111" {
t.Fatalf("bad: %#v", config)
}
if config.ACLAgentMasterToken != "2222" {
t.Fatalf("bad: %#v", config)
}
if config.ACLAgentToken != "3333" {
t.Fatalf("bad: %#v", config)
}
if config.ACLMasterToken != "2345" {
@ -674,6 +694,48 @@ func TestDecodeConfig(t *testing.T) {
t.Fatalf("bad: %#v", config)
}
// ACL token precedence.
input = `{}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
if err != nil {
t.Fatalf("err: %s", err)
}
if token := config.GetTokenForAgent(); token != "" {
t.Fatalf("bad: %s", token)
}
input = `{"acl_token": "hello"}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
if err != nil {
t.Fatalf("err: %s", err)
}
if token := config.GetTokenForAgent(); token != "hello" {
t.Fatalf("bad: %s", token)
}
input = `{"acl_agent_token": "world", "acl_token": "hello"}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
if err != nil {
t.Fatalf("err: %s", err)
}
if token := config.GetTokenForAgent(); token != "world" {
t.Fatalf("bad: %s", token)
}
// ACL flag for Consul version 0.8 features (broken out since we will
// eventually remove this). We first verify this is opt-out.
config = DefaultConfig()
if *config.ACLEnforceVersion8 != false {
t.Fatalf("bad: %#v", config)
}
input = `{"acl_enforce_version_8": true}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
if err != nil {
t.Fatalf("err: %s", err)
}
if *config.ACLEnforceVersion8 != true {
t.Fatalf("bad: %#v", config)
}
// Watches
input = `{"watches": [{"type":"keyprefix", "prefix":"foo/", "handler":"foobar"}]}`
config, err = DecodeConfig(bytes.NewReader([]byte(input)))
@ -1496,6 +1558,9 @@ func TestMergeConfig(t *testing.T) {
DogStatsdAddr: "nope",
DogStatsdTags: []string{"nope"},
},
Meta: map[string]string{
"key": "value1",
},
}
b := &Config{
@ -1570,14 +1635,17 @@ func TestMergeConfig(t *testing.T) {
ReconnectTimeoutWan: 36 * time.Hour,
CheckUpdateInterval: 8 * time.Minute,
CheckUpdateIntervalRaw: "8m",
ACLToken: "1234",
ACLMasterToken: "2345",
ACLToken: "1111",
ACLAgentMasterToken: "2222",
ACLAgentToken: "3333",
ACLMasterToken: "4444",
ACLDatacenter: "dc2",
ACLTTL: 15 * time.Second,
ACLTTLRaw: "15s",
ACLDownPolicy: "deny",
ACLDefaultPolicy: "deny",
ACLReplicationToken: "8765309",
ACLEnforceVersion8: Bool(true),
Watches: []map[string]interface{}{
map[string]interface{}{
"type": "keyprefix",
@ -1594,6 +1662,9 @@ func TestMergeConfig(t *testing.T) {
DogStatsdAddr: "127.0.0.1:7254",
DogStatsdTags: []string{"tag_1:val_1", "tag_2:val_2"},
},
Meta: map[string]string{
"key": "value2",
},
DisableUpdateCheck: true,
DisableAnonymousSignature: true,
HTTPAPIResponseHeaders: map[string]string{

13
command/agent/dns_test.go
View File

@ -569,6 +569,7 @@ func TestDNS_ServiceLookup(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -1021,6 +1022,7 @@ func TestDNS_ServiceLookup_ServiceAddress(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -1115,6 +1117,7 @@ func TestDNS_ServiceLookup_ServiceAddressIPV6(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -1236,6 +1239,7 @@ func TestDNS_ServiceLookup_WanAddress(t *testing.T) {
Datacenter: "dc2",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -1641,6 +1645,7 @@ func TestDNS_ServiceLookup_Dedup(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -1745,6 +1750,7 @@ func TestDNS_ServiceLookup_Dedup_SRV(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -2040,6 +2046,7 @@ func TestDNS_ServiceLookup_FilterCritical(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -2163,6 +2170,7 @@ func TestDNS_ServiceLookup_OnlyFailing(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},
@ -2283,6 +2291,7 @@ func TestDNS_ServiceLookup_OnlyPassing(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
OnlyPassing: true,
@ -2356,6 +2365,7 @@ func TestDNS_ServiceLookup_Randomize(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "web",
},
@ -2450,6 +2460,7 @@ func TestDNS_ServiceLookup_Truncate(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "web",
},
@ -2770,6 +2781,7 @@ func TestDNS_ServiceLookup_CNAME(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "search",
},
@ -4342,6 +4354,7 @@ func TestDNS_Compression_Query(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "db",
},

23
command/agent/event_endpoint.go
View File

@ -83,6 +83,14 @@ func (s *HTTPServer) EventList(resp http.ResponseWriter, req *http.Request) (int
return nil, nil
}
// Fetch the ACL token, if any.
var token string
s.parseToken(req, &token)
acl, err := s.agent.resolveToken(token)
if err != nil {
return nil, err
}
// Look for a name filter
var nameFilter string
if filt := req.URL.Query().Get("name"); filt != "" {
@ -126,7 +134,20 @@ RUN_QUERY:
// Get the recent events
events := s.agent.UserEvents()
// Filter the events if necessary
// Filter the events using the ACL, if present
if acl != nil {
for i := 0; i < len(events); i++ {
name := events[i].Name
if acl.EventRead(name) {
continue
}
s.agent.logger.Printf("[DEBUG] agent: dropping event %q from result due to ACLs", name)
events = append(events[:i], events[i+1:]...)
i--
}
}
// Filter the events if requested
if nameFilter != "" {
for i := 0; i < len(events); i++ {
if events[i].Name != nameFilter {

66
command/agent/event_endpoint_test.go
View File

@ -5,6 +5,7 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
"time"
@ -192,6 +193,71 @@ func TestEventList_Filter(t *testing.T) {
})
}
func TestEventList_ACLFilter(t *testing.T) {
dir, srv := makeHTTPServerWithACLs(t)
defer os.RemoveAll(dir)
defer srv.Shutdown()
defer srv.agent.Shutdown()
// Fire an event.
p := &UserEvent{Name: "foo"}
if err := srv.agent.UserEvent("dc1", "root", p); err != nil {
t.Fatalf("err: %v", err)
}
// Try no token.
{
testutil.WaitForResult(func() (bool, error) {
req, err := http.NewRequest("GET", "/v1/event/list", nil)
if err != nil {
return false, err
}
resp := httptest.NewRecorder()
obj, err := srv.EventList(resp, req)
if err != nil {
return false, err
}
list, ok := obj.([]*UserEvent)
if !ok {
return false, fmt.Errorf("bad: %#v", obj)
}
if len(list) != 0 {
return false, fmt.Errorf("bad: %#v", list)
}
return true, nil
}, func(err error) {
t.Fatalf("err: %v", err)
})
}
// Try the root token.
{
testutil.WaitForResult(func() (bool, error) {
req, err := http.NewRequest("GET", "/v1/event/list?token=root", nil)
if err != nil {
return false, err
}
resp := httptest.NewRecorder()
obj, err := srv.EventList(resp, req)
if err != nil {
return false, err
}
list, ok := obj.([]*UserEvent)
if !ok {
return false, fmt.Errorf("bad: %#v", obj)
}
if len(list) != 1 || list[0].Name != "foo" {
return false, fmt.Errorf("bad: %#v", list)
}
return true, nil
}, func(err error) {
t.Fatalf("err: %v", err)
})
}
}
func TestEventList_Blocking(t *testing.T) {
httpTest(t, func(srv *HTTPServer) {
p := &UserEvent{Name: "test"}

28
command/agent/http.go
View File

@ -241,13 +241,13 @@ func (s *HTTPServer) registerHandlers(enableDebug bool) {
s.handleFuncMetrics("/v1/acl/list", s.wrap(s.ACLList))
s.handleFuncMetrics("/v1/acl/replication", s.wrap(s.ACLReplicationStatus))
} else {
s.handleFuncMetrics("/v1/acl/create", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/update", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/destroy/", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/info/", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/clone/", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/list", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/replication", s.wrap(aclDisabled))
s.handleFuncMetrics("/v1/acl/create", s.wrap(ACLDisabled))
s.handleFuncMetrics("/v1/acl/update", s.wrap(ACLDisabled))
s.handleFuncMetrics("/v1/acl/destroy/", s.wrap(ACLDisabled))
s.handleFuncMetrics("/v1/acl/info/", s.wrap(ACLDisabled))
s.handleFuncMetrics("/v1/acl/clone/", s.wrap(ACLDisabled))
s.handleFuncMetrics("/v1/acl/list", s.wrap(ACLDisabled))
s.handleFuncMetrics("/v1/acl/replication", s.wrap(ACLDisabled))
}
s.handleFuncMetrics("/v1/agent/self", s.wrap(s.AgentSelf))
s.handleFuncMetrics("/v1/agent/maintenance", s.wrap(s.AgentNodeMaintenance))
@ -584,6 +584,20 @@ func (s *HTTPServer) parseSource(req *http.Request, source *structs.QuerySource)
}
}
// parseMetaFilter is used to parse the ?node-meta=key:value query parameter, used for
// filtering results to nodes with the given metadata key/value
func (s *HTTPServer) parseMetaFilter(req *http.Request) map[string]string {
if filterList, ok := req.URL.Query()["node-meta"]; ok {
filters := make(map[string]string)
for _, filter := range filterList {
key, value := parseMetaPair(filter)
filters[key] = value
}
return filters
}
return nil
}
// parse is a convenience method for endpoints that need
// to use both parseWait and parseDC.
func (s *HTTPServer) parse(resp http.ResponseWriter, req *http.Request, dc *string, b *structs.QueryOptions) bool {

16
command/agent/http_test.go
View File

@ -32,6 +32,22 @@ func makeHTTPServerWithConfig(t *testing.T, cb func(c *Config)) (string, *HTTPSe
return makeHTTPServerWithConfigLog(t, cb, nil, nil)
}
func makeHTTPServerWithACLs(t *testing.T) (string, *HTTPServer) {
dir, srv := makeHTTPServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = c.Datacenter
c.ACLDefaultPolicy = "deny"
c.ACLMasterToken = "root"
c.ACLAgentToken = "root"
c.ACLAgentMasterToken = "towel"
c.ACLEnforceVersion8 = Bool(true)
})
// Need a leader to look up ACLs, so wait here so we don't need to
// repeat this in each test.
testutil.WaitForLeader(t, srv.agent.RPC, "dc1")
return dir, srv
}
func makeHTTPServerWithConfigLog(t *testing.T, cb func(c *Config), l io.Writer, logWriter *logger.LogWriter) (string, *HTTPServer) {
configTry := 0
RECONF:

33
command/agent/local.go
View File

@ -18,9 +18,6 @@ import (
const (
syncStaggerIntv = 3 * time.Second
syncRetryIntv = 15 * time.Second
// permissionDenied is returned when an ACL based rejection happens
permissionDenied = "Permission denied"
)
// syncStatus is used to represent the difference between
@ -64,6 +61,9 @@ type localState struct {
// Used to track checks that are being deferred
deferCheck map[types.CheckID]*time.Timer
// metadata tracks the local metadata fields
metadata map[string]string
// consulCh is used to inform of a change to the known
// consul nodes. This may be used to retry a sync run
consulCh chan struct{}
@ -85,6 +85,7 @@ func (l *localState) Init(config *Config, logger *log.Logger) {
l.checkTokens = make(map[types.CheckID]string)
l.checkCriticalTime = make(map[types.CheckID]time.Time)
l.deferCheck = make(map[types.CheckID]*time.Timer)
l.metadata = make(map[string]string)
l.consulCh = make(chan struct{}, 1)
l.triggerCh = make(chan struct{}, 1)
}
@ -342,6 +343,19 @@ func (l *localState) CriticalChecks() map[types.CheckID]CriticalCheck {
return checks
}
// Metadata returns the local node metadata fields that the
// agent is aware of and are being kept in sync with the server
func (l *localState) Metadata() map[string]string {
metadata := make(map[string]string)
l.RLock()
defer l.RUnlock()
for key, value := range l.metadata {
metadata[key] = value
}
return metadata
}
// antiEntropy is a long running method used to perform anti-entropy
// between local and remote state.
func (l *localState) antiEntropy(shutdownCh chan struct{}) {
@ -400,7 +414,7 @@ func (l *localState) setSyncState() error {
req := structs.NodeSpecificRequest{
Datacenter: l.config.Datacenter,
Node: l.config.NodeName,
QueryOptions: structs.QueryOptions{Token: l.config.ACLToken},
QueryOptions: structs.QueryOptions{Token: l.config.GetTokenForAgent()},
}
var out1 structs.IndexedNodeServices
var out2 structs.IndexedHealthChecks
@ -415,10 +429,10 @@ func (l *localState) setSyncState() error {
l.Lock()
defer l.Unlock()
// Check the node info (currently limited to tagged addresses since
// everything else is managed by the Serf layer)
// Check the node info
if out1.NodeServices == nil || out1.NodeServices.Node == nil ||
!reflect.DeepEqual(out1.NodeServices.Node.TaggedAddresses, l.config.TaggedAddresses) {
!reflect.DeepEqual(out1.NodeServices.Node.TaggedAddresses, l.config.TaggedAddresses) ||
!reflect.DeepEqual(out1.NodeServices.Node.Meta, l.metadata) {
l.nodeInfoInSync = false
}
@ -622,6 +636,7 @@ func (l *localState) syncService(id string) error {
Node: l.config.NodeName,
Address: l.config.AdvertiseAddr,
TaggedAddresses: l.config.TaggedAddresses,
NodeMeta: l.metadata,
Service: l.services[id],
WriteRequest: structs.WriteRequest{Token: l.serviceToken(id)},
}
@ -683,6 +698,7 @@ func (l *localState) syncCheck(id types.CheckID) error {
Node: l.config.NodeName,
Address: l.config.AdvertiseAddr,
TaggedAddresses: l.config.TaggedAddresses,
NodeMeta: l.metadata,
Service: service,
Check: l.checks[id],
WriteRequest: structs.WriteRequest{Token: l.checkToken(id)},
@ -709,7 +725,8 @@ func (l *localState) syncNodeInfo() error {
Node: l.config.NodeName,
Address: l.config.AdvertiseAddr,
TaggedAddresses: l.config.TaggedAddresses,
WriteRequest: structs.WriteRequest{Token: l.config.ACLToken},
NodeMeta: l.metadata,
WriteRequest: structs.WriteRequest{Token: l.config.GetTokenForAgent()},
}
var out struct{}
err := l.iface.RPC("Catalog.Register", &req, &out)

7
command/agent/local_test.go
View File

@ -985,6 +985,7 @@ func TestAgentAntiEntropy_Check_DeferSync(t *testing.T) {
func TestAgentAntiEntropy_NodeInfo(t *testing.T) {
conf := nextConfig()
conf.Meta["somekey"] = "somevalue"
dir, agent := makeAgent(t, conf)
defer os.RemoveAll(dir)
defer agent.Shutdown()
@ -1020,7 +1021,8 @@ func TestAgentAntiEntropy_NodeInfo(t *testing.T) {
// Make sure we synced our node info - this should have ridden on the
// "consul" service sync
addrs := services.NodeServices.Node.TaggedAddresses
if len(addrs) == 0 || !reflect.DeepEqual(addrs, conf.TaggedAddresses) {
meta := services.NodeServices.Node.Meta
if len(addrs) == 0 || !reflect.DeepEqual(addrs, conf.TaggedAddresses) || !reflect.DeepEqual(meta, conf.Meta) {
return false, fmt.Errorf("bad: %v", addrs)
}
@ -1044,7 +1046,8 @@ func TestAgentAntiEntropy_NodeInfo(t *testing.T) {
return false, fmt.Errorf("err: %v", err)
}
addrs := services.NodeServices.Node.TaggedAddresses
if len(addrs) == 0 || !reflect.DeepEqual(addrs, conf.TaggedAddresses) {
meta := services.NodeServices.Node.Meta
if len(addrs) == 0 || !reflect.DeepEqual(addrs, conf.TaggedAddresses) || !reflect.DeepEqual(meta, conf.Meta) {
return false, fmt.Errorf("bad: %v", addrs)
}

2
command/agent/session_endpoint.go
View File

@ -46,6 +46,7 @@ func (s *HTTPServer) SessionCreate(resp http.ResponseWriter, req *http.Request)
},
}
s.parseDC(req, &args.Datacenter)
s.parseToken(req, &args.Token)
// Handle optional request body
if req.ContentLength > 0 {
@ -117,6 +118,7 @@ func (s *HTTPServer) SessionDestroy(resp http.ResponseWriter, req *http.Request)
Op: structs.SessionDestroy,
}
s.parseDC(req, &args.Datacenter)
s.parseToken(req, &args.Token)
// Pull out the session id
args.Session.ID = strings.TrimPrefix(req.URL.Path, "/v1/session/destroy/")

123
command/kv_export.go Normal file
View File

@ -0,0 +1,123 @@
package command
import (
"encoding/base64"
"encoding/json"
"flag"
"fmt"
"strings"
"github.com/hashicorp/consul/api"
"github.com/mitchellh/cli"
)
// KVExportCommand is a Command implementation that is used to export
// a KV tree as JSON
type KVExportCommand struct {
Ui cli.Ui
}
func (c *KVExportCommand) Synopsis() string {
return "Exports a tree from the KV store as JSON"
}
func (c *KVExportCommand) Help() string {
helpText := `
Usage: consul kv export [KEY_OR_PREFIX]
Retrieves key-value pairs for the given prefix from Consul's key-value store,
and writes a JSON representation to stdout. This can be used with the command
"consul kv import" to move entire trees between Consul clusters.
$ consul kv export vault
For a full list of options and examples, please see the Consul documentation.
` + apiOptsText + `
KV Export Options:
None.
`
return strings.TrimSpace(helpText)
}
func (c *KVExportCommand) Run(args []string) int {
cmdFlags := flag.NewFlagSet("export", flag.ContinueOnError)
datacenter := cmdFlags.String("datacenter", "", "")
token := cmdFlags.String("token", "", "")
stale := cmdFlags.Bool("stale", false, "")
httpAddr := HTTPAddrFlag(cmdFlags)
if err := cmdFlags.Parse(args); err != nil {
return 1
}
key := ""
// Check for arg validation
args = cmdFlags.Args()
switch len(args) {
case 0:
key = ""
case 1:
key = args[0]
default:
c.Ui.Error(fmt.Sprintf("Too many arguments (expected 1, got %d)", len(args)))
return 1
}
// This is just a "nice" thing to do. Since pairs cannot start with a /, but
// users will likely put "/" or "/foo", lets go ahead and strip that for them
// here.
if len(key) > 0 && key[0] == '/' {
key = key[1:]
}
// Create and test the HTTP client
conf := api.DefaultConfig()
conf.Address = *httpAddr
conf.Token = *token
client, err := api.NewClient(conf)
if err != nil {
c.Ui.Error(fmt.Sprintf("Error connecting to Consul agent: %s", err))
return 1
}
pairs, _, err := client.KV().List(key, &api.QueryOptions{
Datacenter: *datacenter,
AllowStale: *stale,
})
if err != nil {
c.Ui.Error(fmt.Sprintf("Error querying Consul agent: %s", err))
return 1
}
exported := make([]*kvExportEntry, len(pairs))
for i, pair := range pairs {
exported[i] = toExportEntry(pair)
}
marshaled, err := json.MarshalIndent(exported, "", "\t")
if err != nil {
c.Ui.Error(fmt.Sprintf("Error exporting KV data: %s", err))
return 1
}
c.Ui.Info(string(marshaled))
return 0
}
type kvExportEntry struct {
Key string `json:"key"`
Flags uint64 `json:"flags"`
Value string `json:"value"`
}
func toExportEntry(pair *api.KVPair) *kvExportEntry {
return &kvExportEntry{
Key: pair.Key,
Flags: pair.Flags,
Value: base64.StdEncoding.EncodeToString(pair.Value),
}
}

60
command/kv_export_test.go Normal file
View File

@ -0,0 +1,60 @@
package command
import (
"encoding/base64"
"encoding/json"
"testing"
"github.com/hashicorp/consul/api"
"github.com/mitchellh/cli"
)
func TestKVExportCommand_Run(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()
waitForLeader(t, srv.httpAddr)
ui := new(cli.MockUi)
c := &KVExportCommand{Ui: ui}
keys := map[string]string{
"foo/a": "a",
"foo/b": "b",
"foo/c": "c",
"bar": "d",
}
for k, v := range keys {
pair := &api.KVPair{Key: k, Value: []byte(v)}
if _, err := client.KV().Put(pair, nil); err != nil {
t.Fatalf("err: %#v", err)
}
}
args := []string{
"-http-addr=" + srv.httpAddr,
"foo",
}
code := c.Run(args)
if code != 0 {
t.Fatalf("bad: %d. %#v", code, ui.ErrorWriter.String())
}
output := ui.OutputWriter.String()
var exported []*kvExportEntry
err := json.Unmarshal([]byte(output), &exported)
if err != nil {
t.Fatalf("bad: %d", code)
}
if len(exported) != 3 {
t.Fatalf("bad: expected 3, got %d", len(exported))
}
for _, entry := range exported {
if base64.StdEncoding.EncodeToString([]byte(keys[entry.Key])) != entry.Value {
t.Fatalf("bad: expected %s, got %s", keys[entry.Key], entry.Value)
}
}
}

24
command/kv_get.go
View File

@ -2,6 +2,7 @@ package command
import (
"bytes"
"encoding/base64"
"flag"
"fmt"
"io"
@ -54,6 +55,8 @@ Usage: consul kv get [options] [KEY_OR_PREFIX]
KV Get Options:
-base64 Base64 encode the value. The default value is false.
-detailed Provide additional metadata about the key in addition
to the value such as the ModifyIndex and any flags
that may have been set on the key. The default value
@ -84,6 +87,7 @@ func (c *KVGetCommand) Run(args []string) int {
stale := cmdFlags.Bool("stale", false, "")
detailed := cmdFlags.Bool("detailed", false, "")
keys := cmdFlags.Bool("keys", false, "")
base64encode := cmdFlags.Bool("base64", false, "")
recurse := cmdFlags.Bool("recurse", false, "")
separator := cmdFlags.String("separator", "/", "")
httpAddr := HTTPAddrFlag(cmdFlags)
@ -158,7 +162,7 @@ func (c *KVGetCommand) Run(args []string) int {
for i, pair := range pairs {
if *detailed {
var b bytes.Buffer
if err := prettyKVPair(&b, pair); err != nil {
if err := prettyKVPair(&b, pair, *base64encode); err != nil {
c.Ui.Error(fmt.Sprintf("Error rendering KV pair: %s", err))
return 1
}
@ -169,7 +173,11 @@ func (c *KVGetCommand) Run(args []string) int {
c.Ui.Info("")
}
} else {
c.Ui.Info(fmt.Sprintf("%s:%s", pair.Key, pair.Value))
if *base64encode {
c.Ui.Info(fmt.Sprintf("%s:%s", pair.Key, base64.StdEncoding.EncodeToString(pair.Value)))
} else {
c.Ui.Info(fmt.Sprintf("%s:%s", pair.Key, pair.Value))
}
}
}
@ -191,7 +199,7 @@ func (c *KVGetCommand) Run(args []string) int {
if *detailed {
var b bytes.Buffer
if err := prettyKVPair(&b, pair); err != nil {
if err := prettyKVPair(&b, pair, *base64encode); err != nil {
c.Ui.Error(fmt.Sprintf("Error rendering KV pair: %s", err))
return 1
}
@ -209,7 +217,7 @@ func (c *KVGetCommand) Synopsis() string {
return "Retrieves or lists data from the KV store"
}
func prettyKVPair(w io.Writer, pair *api.KVPair) error {
func prettyKVPair(w io.Writer, pair *api.KVPair, base64EncodeValue bool) error {
tw := tabwriter.NewWriter(w, 0, 2, 6, ' ', 0)
fmt.Fprintf(tw, "CreateIndex\t%d\n", pair.CreateIndex)
fmt.Fprintf(tw, "Flags\t%d\n", pair.Flags)
@ -217,10 +225,14 @@ func prettyKVPair(w io.Writer, pair *api.KVPair) error {
fmt.Fprintf(tw, "LockIndex\t%d\n", pair.LockIndex)
fmt.Fprintf(tw, "ModifyIndex\t%d\n", pair.ModifyIndex)
if pair.Session == "" {
fmt.Fprintf(tw, "Session\t-\n")
fmt.Fprint(tw, "Session\t-\n")
} else {
fmt.Fprintf(tw, "Session\t%s\n", pair.Session)
}
fmt.Fprintf(tw, "Value\t%s", pair.Value)
if base64EncodeValue {
fmt.Fprintf(tw, "Value\t%s", base64.StdEncoding.EncodeToString(pair.Value))
} else {
fmt.Fprintf(tw, "Value\t%s", pair.Value)
}
return tw.Flush()
}

89
command/kv_get_test.go
View File

@ -1,6 +1,7 @@
package command
import (
"encoding/base64"
"strings"
"testing"
@ -250,3 +251,91 @@ func TestKVGetCommand_Recurse(t *testing.T) {
}
}
}
func TestKVGetCommand_RecurseBase64(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()
waitForLeader(t, srv.httpAddr)
ui := new(cli.MockUi)
c := &KVGetCommand{Ui: ui}
keys := map[string]string{
"foo/a": "Hello World 1",
"foo/b": "Hello World 2",
"foo/c": "Hello World 3",
}
for k, v := range keys {
pair := &api.KVPair{Key: k, Value: []byte(v)}
if _, err := client.KV().Put(pair, nil); err != nil {
t.Fatalf("err: %#v", err)
}
}
args := []string{
"-http-addr=" + srv.httpAddr,
"-recurse",
"-base64",
"foo",
}
code := c.Run(args)
if code != 0 {
t.Fatalf("bad: %d. %#v", code, ui.ErrorWriter.String())
}
output := ui.OutputWriter.String()
for key, value := range keys {
if !strings.Contains(output, key+":"+base64.StdEncoding.EncodeToString([]byte(value))) {
t.Fatalf("bad %#v missing %q", output, key)
}
}
}
func TestKVGetCommand_DetailedBase64(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()
waitForLeader(t, srv.httpAddr)
ui := new(cli.MockUi)
c := &KVGetCommand{Ui: ui}
pair := &api.KVPair{
Key: "foo",
Value: []byte("bar"),
}
_, err := client.KV().Put(pair, nil)
if err != nil {
t.Fatalf("err: %#v", err)
}
args := []string{
"-http-addr=" + srv.httpAddr,
"-detailed",
"-base64",
"foo",
}
code := c.Run(args)
if code != 0 {
t.Fatalf("bad: %d. %#v", code, ui.ErrorWriter.String())
}
output := ui.OutputWriter.String()
for _, key := range []string{
"CreateIndex",
"LockIndex",
"ModifyIndex",
"Flags",
"Session",
"Value",
} {
if !strings.Contains(output, key) {
t.Fatalf("bad %#v, missing %q", output, key)
}
}
if !strings.Contains(output, base64.StdEncoding.EncodeToString([]byte("bar"))) {
t.Fatalf("bad %#v, value is not base64 encoded", output)
}
}

165
command/kv_import.go Normal file
View File

@ -0,0 +1,165 @@
package command
import (
"bytes"
"encoding/base64"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"io/ioutil"
"os"
"strings"
"github.com/hashicorp/consul/api"
"github.com/mitchellh/cli"
)
// KVImportCommand is a Command implementation that is used to import
// a KV tree stored as JSON
type KVImportCommand struct {
Ui cli.Ui
// testStdin is the input for testing.
testStdin io.Reader
}
func (c *KVImportCommand) Synopsis() string {
return "Imports a tree stored as JSON to the KV store"
}
func (c *KVImportCommand) Help() string {
helpText := `
Usage: consul kv import [DATA]
Imports key-value pairs to the key-value store from the JSON representation
generated by the "consul kv export" command.
The data can be read from a file by prefixing the filename with the "@"
symbol. For example:
$ consul kv import @filename.json
Or it can be read from stdin using the "-" symbol:
$ cat filename.json | consul kv import config/program/license -
Alternatively the data may be provided as the final parameter to the command,
though care must be taken with regards to shell escaping.
For a full list of options and examples, please see the Consul documentation.
` + apiOptsText + `
KV Import Options:
None.
`
return strings.TrimSpace(helpText)
}
func (c *KVImportCommand) Run(args []string) int {
cmdFlags := flag.NewFlagSet("import", flag.ContinueOnError)
datacenter := cmdFlags.String("datacenter", "", "")
token := cmdFlags.String("token", "", "")
httpAddr := HTTPAddrFlag(cmdFlags)
if err := cmdFlags.Parse(args); err != nil {
return 1
}
// Check for arg validation
args = cmdFlags.Args()
data, err := c.dataFromArgs(args)
if err != nil {
c.Ui.Error(fmt.Sprintf("Error! %s", err))
return 1
}
// Create and test the HTTP client
conf := api.DefaultConfig()
conf.Address = *httpAddr
conf.Token = *token
client, err := api.NewClient(conf)
if err != nil {
c.Ui.Error(fmt.Sprintf("Error connecting to Consul agent: %s", err))
return 1
}
var entries []*kvExportEntry
if err := json.Unmarshal([]byte(data), &entries); err != nil {
c.Ui.Error(fmt.Sprintf("Cannot unmarshal data: %s", err))
return 1
}
for _, entry := range entries {
value, err := base64.StdEncoding.DecodeString(entry.Value)
if err != nil {
c.Ui.Error(fmt.Sprintf("Error base 64 decoding value for key %s: %s", entry.Key, err))
return 1
}
pair := &api.KVPair{
Key: entry.Key,
Flags: entry.Flags,
Value: value,
}
wo := &api.WriteOptions{
Datacenter: *datacenter,
Token: *token,
}
if _, err := client.KV().Put(pair, wo); err != nil {
c.Ui.Error(fmt.Sprintf("Error! Failed writing data for key %s: %s", pair.Key, err))
return 1
}
c.Ui.Info(fmt.Sprintf("Imported: %s", pair.Key))
}
return 0
}
func (c *KVImportCommand) dataFromArgs(args []string) (string, error) {
var stdin io.Reader = os.Stdin
if c.testStdin != nil {
stdin = c.testStdin
}
switch len(args) {
case 0:
return "", errors.New("Missing DATA argument")
case 1:
default:
return "", fmt.Errorf("Too many arguments (expected 1 or 2, got %d)", len(args))
}
data := args[0]
if len(data) == 0 {
return "", errors.New("Empty DATA argument")
}
switch data[0] {
case '@':
data, err := ioutil.ReadFile(data[1:])
if err != nil {
return "", fmt.Errorf("Failed to read file: %s", err)
}
return string(data), nil
case '-':
if len(data) > 1 {
return data, nil
} else {
var b bytes.Buffer
if _, err := io.Copy(&b, stdin); err != nil {
return "", fmt.Errorf("Failed to read stdin: %s", err)
}
return b.String(), nil
}
default:
return data, nil
}
}

61
command/kv_import_test.go Normal file
View File

@ -0,0 +1,61 @@
package command
import (
"strings"
"testing"
"github.com/mitchellh/cli"
)
func TestKVImportCommand_Run(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()
waitForLeader(t, srv.httpAddr)
const json = `[
{
"key": "foo",
"flags": 0,
"value": "YmFyCg=="
},
{
"key": "foo/a",
"flags": 0,
"value": "YmF6Cg=="
}
]`
ui := new(cli.MockUi)
c := &KVImportCommand{
Ui: ui,
testStdin: strings.NewReader(json),
}
args := []string{
"-http-addr=" + srv.httpAddr,
"-",
}
code := c.Run(args)
if code != 0 {
t.Fatalf("bad: %d. %#v", code, ui.ErrorWriter.String())
}
pair, _, err := client.KV().Get("foo", nil)
if err != nil {
t.Fatal(err)
}
if strings.TrimSpace(string(pair.Value)) != "bar" {
t.Fatalf("bad: expected: bar, got %s", pair.Value)
}
pair, _, err = client.KV().Get("foo/a", nil)
if err != nil {
t.Fatal(err)
}
if strings.TrimSpace(string(pair.Value)) != "baz" {
t.Fatalf("bad: expected: baz, got %s", pair.Value)
}
}

23
command/kv_put.go
View File

@ -2,6 +2,7 @@ package command
import (
"bytes"
"encoding/base64"
"flag"
"fmt"
"io"
@ -45,6 +46,9 @@ Usage: consul kv put [options] KEY [DATA]
$ consul kv put webapp/beta/active
If the -base64 flag is specified, the data will be treated as base 64
encoded.
To perform a Check-And-Set operation, specify the -cas flag with the
appropriate -modify-index flag corresponding to the key you want to perform
the CAS operation on:
@ -62,6 +66,9 @@ KV Put Options:
lock. The session must already exist and be specified
via the -session flag. The default value is false.
-base64 Treat the data as base 64 encoded. The default value
is false.
-cas Perform a Check-And-Set operation. Specifying this
value also requires the -modify-index flag to be set.
The default value is false.
@ -95,6 +102,7 @@ func (c *KVPutCommand) Run(args []string) int {
token := cmdFlags.String("token", "", "")
cas := cmdFlags.Bool("cas", false, "")
flags := cmdFlags.Uint64("flags", 0, "")
base64encoded := cmdFlags.Bool("base64", false, "")
modifyIndex := cmdFlags.Uint64("modify-index", 0, "")
session := cmdFlags.String("session", "", "")
acquire := cmdFlags.Bool("acquire", false, "")
@ -111,6 +119,14 @@ func (c *KVPutCommand) Run(args []string) int {
return 1
}
dataBytes := []byte(data)
if *base64encoded {
dataBytes, err = base64.StdEncoding.DecodeString(data)
if err != nil {
c.Ui.Error(fmt.Sprintf("Error! Cannot base 64 decode data: %s", err))
}
}
// Session is reauired for release or acquire
if (*release || *acquire) && *session == "" {
c.Ui.Error("Error! Missing -session (required with -acquire and -release)")
@ -137,7 +153,7 @@ func (c *KVPutCommand) Run(args []string) int {
Key: key,
ModifyIndex: *modifyIndex,
Flags: *flags,
Value: []byte(data),
Value: dataBytes,
Session: *session,
}
@ -220,6 +236,11 @@ func (c *KVPutCommand) dataFromArgs(args []string) (string, string, error) {
key := args[0]
data := args[1]
// Handle empty quoted shell parameters
if len(data) == 0 {
return key, "", nil
}
switch data[0] {
case '@':
data, err := ioutil.ReadFile(data[1:])

65
command/kv_put_test.go
View File

@ -2,6 +2,7 @@ package command
import (
"bytes"
"encoding/base64"
"io"
"io/ioutil"
"os"
@ -100,6 +101,70 @@ func TestKVPutCommand_Run(t *testing.T) {
}
}
func TestKVPutCommand_RunEmptyDataQuoted(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()
waitForLeader(t, srv.httpAddr)
ui := new(cli.MockUi)
c := &KVPutCommand{Ui: ui}
args := []string{
"-http-addr=" + srv.httpAddr,
"foo", "",
}
code := c.Run(args)
if code != 0 {
t.Fatalf("bad: %d. %#v", code, ui.ErrorWriter.String())
}
data, _, err := client.KV().Get("foo", nil)
if err != nil {
t.Fatal(err)
}
if data.Value != nil {
t.Errorf("bad: %#v", data.Value)
}
}
func TestKVPutCommand_RunBase64(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()
waitForLeader(t, srv.httpAddr)
ui := new(cli.MockUi)
c := &KVPutCommand{Ui: ui}
const encodedString = "aGVsbG8gd29ybGQK"
args := []string{
"-http-addr=" + srv.httpAddr,
"-base64",
"foo", encodedString,
}
code := c.Run(args)
if code != 0 {
t.Fatalf("bad: %d. %#v", code, ui.ErrorWriter.String())
}
data, _, err := client.KV().Get("foo", nil)
if err != nil {
t.Fatal(err)
}
expected, err := base64.StdEncoding.DecodeString(encodedString)
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(data.Value, []byte(expected)) {
t.Errorf("bad: %#v, %s", data.Value, data.Value)
}
}
func TestKVPutCommand_File(t *testing.T) {
srv, client := testAgentWithAPIClient(t)
defer srv.Shutdown()

12
commands.go
View File

@ -78,6 +78,18 @@ func init() {
}, nil
},
"kv export": func() (cli.Command, error) {
return &command.KVExportCommand{
Ui: ui,
}, nil
},
"kv import": func() (cli.Command, error) {
return &command.KVImportCommand{
Ui: ui,
}, nil
},
"join": func() (cli.Command, error) {
return &command.JoinCommand{
Ui: ui,

340
consul/acl.go
View File

@ -14,29 +14,30 @@ import (
"github.com/hashicorp/golang-lru"
)
// These must be kept in sync with the constants in command/agent/acl.go.
const (
// aclNotFound indicates there is no matching ACL
// aclNotFound indicates there is no matching ACL.
aclNotFound = "ACL not found"
// rootDenied is returned when attempting to resolve a root ACL
// rootDenied is returned when attempting to resolve a root ACL.
rootDenied = "Cannot resolve root ACL"
// permissionDenied is returned when an ACL based rejection happens
// permissionDenied is returned when an ACL based rejection happens.
permissionDenied = "Permission denied"
// aclDisabled is returned when ACL changes are not permitted
// since they are disabled.
// aclDisabled is returned when ACL changes are not permitted since they
// are disabled.
aclDisabled = "ACL support disabled"
// anonymousToken is the token ID we re-write to if there
// is no token ID provided
// anonymousToken is the token ID we re-write to if there is no token ID
// provided.
anonymousToken = "anonymous"
// redactedToken is shown in structures with embedded tokens when they
// are not allowed to be displayed
// are not allowed to be displayed.
redactedToken = "<hidden>"
// Maximum number of cached ACL entries
// Maximum number of cached ACL entries.
aclCacheSize = 10 * 1024
)
@ -264,6 +265,8 @@ func (c *aclCache) useACLPolicy(id, authDC string, cached *aclCacheEntry, p *str
// Check if we can used the cached policy
if cached != nil && cached.ETag == p.ETag {
if p.TTL > 0 {
// TODO (slackpad) - This seems like it's an unsafe
// write.
cached.Expires = time.Now().Add(p.TTL)
}
return cached.ACL, nil
@ -311,33 +314,55 @@ func (c *aclCache) useACLPolicy(id, authDC string, cached *aclCacheEntry, p *str
// aclFilter is used to filter results from our state store based on ACL rules
// configured for the provided token.
type aclFilter struct {
acl acl.ACL
logger *log.Logger
acl acl.ACL
logger *log.Logger
enforceVersion8 bool
}
// newAclFilter constructs a new aclFilter.
func newAclFilter(acl acl.ACL, logger *log.Logger) *aclFilter {
func newAclFilter(acl acl.ACL, logger *log.Logger, enforceVersion8 bool) *aclFilter {
if logger == nil {
logger = log.New(os.Stdout, "", log.LstdFlags)
}
return &aclFilter{acl, logger}
return &aclFilter{
acl: acl,
logger: logger,
enforceVersion8: enforceVersion8,
}
}
// filterService is used to determine if a service is accessible for an ACL.
func (f *aclFilter) filterService(service string) bool {
// allowNode is used to determine if a node is accessible for an ACL.
func (f *aclFilter) allowNode(node string) bool {
if !f.enforceVersion8 {
return true
}
return f.acl.NodeRead(node)
}
// allowService is used to determine if a service is accessible for an ACL.
func (f *aclFilter) allowService(service string) bool {
if service == "" || service == ConsulServiceID {
return true
}
return f.acl.ServiceRead(service)
}
// allowSession is used to determine if a session for a node is accessible for
// an ACL.
func (f *aclFilter) allowSession(node string) bool {
if !f.enforceVersion8 {
return true
}
return f.acl.SessionRead(node)
}
// filterHealthChecks is used to filter a set of health checks down based on
// the configured ACL rules for a token.
func (f *aclFilter) filterHealthChecks(checks *structs.HealthChecks) {
hc := *checks
for i := 0; i < len(hc); i++ {
check := hc[i]
if f.filterService(check.ServiceName) {
if f.allowNode(check.Node) && f.allowService(check.ServiceName) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping check %q from result due to ACLs", check.CheckID)
@ -350,7 +375,7 @@ func (f *aclFilter) filterHealthChecks(checks *structs.HealthChecks) {
// filterServices is used to filter a set of services based on ACLs.
func (f *aclFilter) filterServices(services structs.Services) {
for svc, _ := range services {
if f.filterService(svc) {
if f.allowService(svc) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping service %q from result due to ACLs", svc)
@ -364,7 +389,7 @@ func (f *aclFilter) filterServiceNodes(nodes *structs.ServiceNodes) {
sn := *nodes
for i := 0; i < len(sn); i++ {
node := sn[i]
if f.filterService(node.ServiceName) {
if f.allowNode(node.Node) && f.allowService(node.ServiceName) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping node %q from result due to ACLs", node.Node)
@ -375,13 +400,22 @@ func (f *aclFilter) filterServiceNodes(nodes *structs.ServiceNodes) {
}
// filterNodeServices is used to filter services on a given node base on ACLs.
func (f *aclFilter) filterNodeServices(services *structs.NodeServices) {
for svc, _ := range services.Services {
if f.filterService(svc) {
func (f *aclFilter) filterNodeServices(services **structs.NodeServices) {
if *services == nil {
return
}
if !f.allowNode((*services).Node.Node) {
*services = nil
return
}
for svc, _ := range (*services).Services {
if f.allowService(svc) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping service %q from result due to ACLs", svc)
delete(services.Services, svc)
delete((*services).Services, svc)
}
}
@ -390,7 +424,7 @@ func (f *aclFilter) filterCheckServiceNodes(nodes *structs.CheckServiceNodes) {
csn := *nodes
for i := 0; i < len(csn); i++ {
node := csn[i]
if f.filterService(node.Service.Service) {
if f.allowNode(node.Node.Node) && f.allowService(node.Service.Service) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping node %q from result due to ACLs", node.Node.Node)
@ -400,6 +434,37 @@ func (f *aclFilter) filterCheckServiceNodes(nodes *structs.CheckServiceNodes) {
*nodes = csn
}
// filterSessions is used to filter a set of sessions based on ACLs.
func (f *aclFilter) filterSessions(sessions *structs.Sessions) {
s := *sessions
for i := 0; i < len(s); i++ {
session := s[i]
if f.allowSession(session.Node) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping session %q from result due to ACLs", session.ID)
s = append(s[:i], s[i+1:]...)
i--
}
*sessions = s
}
// filterCoordinates is used to filter nodes in a coordinate dump based on ACL
// rules.
func (f *aclFilter) filterCoordinates(coords *structs.Coordinates) {
c := *coords
for i := 0; i < len(c); i++ {
node := c[i].Node
if f.allowNode(node) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping node %q from result due to ACLs", node)
c = append(c[:i], c[i+1:]...)
i--
}
*coords = c
}
// filterNodeDump is used to filter through all parts of a node dump and
// remove elements the provided ACL token cannot access.
func (f *aclFilter) filterNodeDump(dump *structs.NodeDump) {
@ -407,31 +472,55 @@ func (f *aclFilter) filterNodeDump(dump *structs.NodeDump) {
for i := 0; i < len(nd); i++ {
info := nd[i]
// Filter nodes
if node := info.Node; !f.allowNode(node) {
f.logger.Printf("[DEBUG] consul: dropping node %q from result due to ACLs", node)
nd = append(nd[:i], nd[i+1:]...)
i--
continue
}
// Filter services
for i := 0; i < len(info.Services); i++ {
svc := info.Services[i].Service
if f.filterService(svc) {
for j := 0; j < len(info.Services); j++ {
svc := info.Services[j].Service
if f.allowService(svc) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping service %q from result due to ACLs", svc)
info.Services = append(info.Services[:i], info.Services[i+1:]...)
i--
info.Services = append(info.Services[:j], info.Services[j+1:]...)
j--
}
// Filter checks
for i := 0; i < len(info.Checks); i++ {
chk := info.Checks[i]
if f.filterService(chk.ServiceName) {
for j := 0; j < len(info.Checks); j++ {
chk := info.Checks[j]
if f.allowService(chk.ServiceName) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping check %q from result due to ACLs", chk.CheckID)
info.Checks = append(info.Checks[:i], info.Checks[i+1:]...)
i--
info.Checks = append(info.Checks[:j], info.Checks[j+1:]...)
j--
}
}
*dump = nd
}
// filterNodes is used to filter through all parts of a node list and remove
// elements the provided ACL token cannot access.
func (f *aclFilter) filterNodes(nodes *structs.Nodes) {
n := *nodes
for i := 0; i < len(n); i++ {
node := n[i].Node
if f.allowNode(node) {
continue
}
f.logger.Printf("[DEBUG] consul: dropping node %q from result due to ACLs", node)
n = append(n[:i], n[i+1:]...)
i--
}
*nodes = n
}
// redactPreparedQueryTokens will redact any tokens unless the client has a
// management token. This eases the transition to delegated authority over
// prepared queries, since it was easy to capture management tokens in Consul
@ -506,32 +595,39 @@ func (s *Server) filterACL(token string, subj interface{}) error {
}
// Create the filter
filt := newAclFilter(acl, s.logger)
filt := newAclFilter(acl, s.logger, s.config.ACLEnforceVersion8)
switch v := subj.(type) {
case *structs.IndexedHealthChecks:
filt.filterHealthChecks(&v.HealthChecks)
case *structs.IndexedServices:
filt.filterServices(v.Services)
case *structs.IndexedServiceNodes:
filt.filterServiceNodes(&v.ServiceNodes)
case *structs.IndexedNodeServices:
if v.NodeServices != nil {
filt.filterNodeServices(v.NodeServices)
}
case *structs.CheckServiceNodes:
filt.filterCheckServiceNodes(v)
case *structs.IndexedCheckServiceNodes:
filt.filterCheckServiceNodes(&v.Nodes)
case *structs.CheckServiceNodes:
filt.filterCheckServiceNodes(v)
case *structs.IndexedCoordinates:
filt.filterCoordinates(&v.Coordinates)
case *structs.IndexedHealthChecks:
filt.filterHealthChecks(&v.HealthChecks)
case *structs.IndexedNodeDump:
filt.filterNodeDump(&v.Dump)
case *structs.IndexedNodes:
filt.filterNodes(&v.Nodes)
case *structs.IndexedNodeServices:
filt.filterNodeServices(&v.NodeServices)
case *structs.IndexedServiceNodes:
filt.filterServiceNodes(&v.ServiceNodes)
case *structs.IndexedServices:
filt.filterServices(v.Services)
case *structs.IndexedSessions:
filt.filterSessions(&v.Sessions)
case *structs.IndexedPreparedQueries:
filt.filterPreparedQueries(&v.Queries)
@ -544,3 +640,149 @@ func (s *Server) filterACL(token string, subj interface{}) error {
return nil
}
// vetRegisterWithACL applies the given ACL's policy to the catalog update and
// determines if it is allowed. Since the catalog register request is so
// dynamic, this is a pretty complex algorithm and was worth breaking out of the
// endpoint. The NodeServices record for the node must be supplied, and can be
// nil.
//
// This is a bit racy because we have to check the state store outside of a
// transaction. It's the best we can do because we don't want to flow ACL
// checking down there. The node information doesn't change in practice, so this
// will be fine. If we expose ways to change node addresses in a later version,
// then we should split the catalog API at the node and service level so we can
// address this race better (even then it would be super rare, and would at
// worst let a service update revert a recent node update, so it doesn't open up
// too much abuse).
func vetRegisterWithACL(acl acl.ACL, subj *structs.RegisterRequest,
ns *structs.NodeServices) error {
// Fast path if ACLs are not enabled.
if acl == nil {
return nil
}
// Vet the node info. This allows service updates to re-post the required
// node info for each request without having to have node "write"
// privileges.
needsNode := ns == nil || subj.ChangesNode(ns.Node)
if needsNode && !acl.NodeWrite(subj.Node) {
return permissionDeniedErr
}
// Vet the service change. This includes making sure they can register
// the given service, and that we can write to any existing service that
// is being modified by id (if any).
if subj.Service != nil {
if !acl.ServiceWrite(subj.Service.Service) {
return permissionDeniedErr
}
if ns != nil {
other, ok := ns.Services[subj.Service.ID]
if ok && !acl.ServiceWrite(other.Service) {
return permissionDeniedErr
}
}
}
// Make sure that the member was flattened before we got there. This
// keeps us from having to verify this check as well.
if subj.Check != nil {
return fmt.Errorf("check member must be nil")
}
// Vet the checks. Node-level checks require node write, and
// service-level checks require service write.
for _, check := range subj.Checks {
// Make sure that the node matches - we don't allow you to mix
// checks from other nodes because we'd have to pull a bunch
// more state store data to check this. If ACLs are enabled then
// we simply require them to match in a given request. There's a
// note in state_store.go to ban this down there in Consul 0.8,
// but it's good to leave this here because it's required for
// correctness wrt. ACLs.
if check.Node != subj.Node {
return fmt.Errorf("Node '%s' for check '%s' doesn't match register request node '%s'",
check.Node, check.CheckID, subj.Node)
}
// Node-level check.
if check.ServiceID == "" {
if !acl.NodeWrite(subj.Node) {
return permissionDeniedErr
}
continue
}
// Service-level check, check the common case where it
// matches the service part of this request, which has
// already been vetted above, and might be being registered
// along with its checks.
if subj.Service != nil && subj.Service.ID == check.ServiceID {
continue
}
// Service-level check for some other service. Make sure they've
// got write permissions for that service.
if ns == nil {
return fmt.Errorf("Unknown service '%s' for check '%s'",
check.ServiceID, check.CheckID)
} else {
other, ok := ns.Services[check.ServiceID]
if !ok {
return fmt.Errorf("Unknown service '%s' for check '%s'",
check.ServiceID, check.CheckID)
}
if !acl.ServiceWrite(other.Service) {
return permissionDeniedErr
}
}
}
return nil
}
// vetDeregisterWithACL applies the given ACL's policy to the catalog update and
// determines if it is allowed. Since the catalog deregister request is so
// dynamic, this is a pretty complex algorithm and was worth breaking out of the
// endpoint. The NodeService for the referenced service must be supplied, and can
// be nil; similar for the HealthCheck for the referenced health check.
func vetDeregisterWithACL(acl acl.ACL, subj *structs.DeregisterRequest,
ns *structs.NodeService, nc *structs.HealthCheck) error {
// Fast path if ACLs are not enabled.
if acl == nil {
return nil
}
// This order must match the code in applyRegister() in fsm.go since it
// also evaluates things in this order, and will ignore fields based on
// this precedence. This lets us also ignore them from an ACL perspective.
if subj.ServiceID != "" {
if ns == nil {
return fmt.Errorf("Unknown service '%s'", subj.ServiceID)
}
if !acl.ServiceWrite(ns.Service) {
return permissionDeniedErr
}
} else if subj.CheckID != "" {
if nc == nil {
return fmt.Errorf("Unknown check '%s'", subj.CheckID)
}
if nc.ServiceID != "" {
if !acl.ServiceWrite(nc.ServiceName) {
return permissionDeniedErr
}
} else {
if !acl.NodeWrite(subj.Node) {
return permissionDeniedErr
}
}
} else {
if !acl.NodeWrite(subj.Node) {
return permissionDeniedErr
}
}
return nil
}

1075
consul/acl_test.go
View File

File diff suppressed because it is too large Load Diff

97
consul/catalog_endpoint.go
View File

@ -21,37 +21,42 @@ func (c *Catalog) Register(args *structs.RegisterRequest, reply *struct{}) error
}
defer metrics.MeasureSince([]string{"consul", "catalog", "register"}, time.Now())
// Verify the args
// Verify the args.
if args.Node == "" || args.Address == "" {
return fmt.Errorf("Must provide node and address")
}
// Fetch the ACL token, if any.
acl, err := c.srv.resolveToken(args.Token)
if err != nil {
return err
}
// Handle a service registration.
if args.Service != nil {
// If no service id, but service name, use default
if args.Service.ID == "" && args.Service.Service != "" {
args.Service.ID = args.Service.Service
}
// Verify ServiceName provided if ID
// Verify ServiceName provided if ID.
if args.Service.ID != "" && args.Service.Service == "" {
return fmt.Errorf("Must provide service name with ID")
}
// Apply the ACL policy if any
// The 'consul' service is excluded since it is managed
// automatically internally.
// Apply the ACL policy if any. The 'consul' service is excluded
// since it is managed automatically internally (that behavior
// is going away after version 0.8). We check this same policy
// later if version 0.8 is enabled, so we can eventually just
// delete this and do all the ACL checks down there.
if args.Service.Service != ConsulServiceName {
acl, err := c.srv.resolveToken(args.Token)
if err != nil {
return err
} else if acl != nil && !acl.ServiceWrite(args.Service.Service) {
c.srv.logger.Printf("[WARN] consul.catalog: Register of service '%s' on '%s' denied due to ACLs",
args.Service.Service, args.Node)
if acl != nil && !acl.ServiceWrite(args.Service.Service) {
return permissionDeniedErr
}
}
}
// Move the old format single check into the slice, and fixup IDs.
if args.Check != nil {
args.Checks = append(args.Checks, args.Check)
args.Check = nil
@ -65,9 +70,20 @@ func (c *Catalog) Register(args *structs.RegisterRequest, reply *struct{}) error
}
}
_, err := c.srv.raftApply(structs.RegisterRequestType, args)
// Check the complete register request against the given ACL policy.
if acl != nil && c.srv.config.ACLEnforceVersion8 {
state := c.srv.fsm.State()
_, ns, err := state.NodeServices(args.Node)
if err != nil {
return fmt.Errorf("Node lookup failed: %v", err)
}
if err := vetRegisterWithACL(acl, args, ns); err != nil {
return err
}
}
_, err = c.srv.raftApply(structs.RegisterRequestType, args)
if err != nil {
c.srv.logger.Printf("[ERR] consul.catalog: Register failed: %v", err)
return err
}
@ -86,9 +102,38 @@ func (c *Catalog) Deregister(args *structs.DeregisterRequest, reply *struct{}) e
return fmt.Errorf("Must provide node")
}
_, err := c.srv.raftApply(structs.DeregisterRequestType, args)
// Fetch the ACL token, if any.
acl, err := c.srv.resolveToken(args.Token)
if err != nil {
c.srv.logger.Printf("[ERR] consul.catalog: Deregister failed: %v", err)
return err
}
// Check the complete deregister request against the given ACL policy.
if acl != nil && c.srv.config.ACLEnforceVersion8 {
state := c.srv.fsm.State()
var ns *structs.NodeService
if args.ServiceID != "" {
_, ns, err = state.NodeService(args.Node, args.ServiceID)
if err != nil {
return fmt.Errorf("Service lookup failed: %v", err)
}
}
var nc *structs.HealthCheck
if args.CheckID != "" {
_, nc, err = state.NodeCheck(args.Node, args.CheckID)
if err != nil {
return fmt.Errorf("Check lookup failed: %v", err)
}
}
if err := vetDeregisterWithACL(acl, args, ns, nc); err != nil {
return err
}
}
if _, err := c.srv.raftApply(structs.DeregisterRequestType, args); err != nil {
return err
}
return nil
@ -118,12 +163,22 @@ func (c *Catalog) ListNodes(args *structs.DCSpecificRequest, reply *structs.Inde
&reply.QueryMeta,
state.GetQueryWatch("Nodes"),
func() error {
index, nodes, err := state.Nodes()
var index uint64
var nodes structs.Nodes
var err error
if len(args.NodeMetaFilters) > 0 {
index, nodes, err = state.NodesByMeta(args.NodeMetaFilters)
} else {
index, nodes, err = state.Nodes()
}
if err != nil {
return err
}
reply.Index, reply.Nodes = index, nodes
if err := c.srv.filterACL(args.Token, reply); err != nil {
return err
}
return c.srv.sortNodesByDistanceFrom(args.Source, reply.Nodes)
})
}
@ -141,7 +196,14 @@ func (c *Catalog) ListServices(args *structs.DCSpecificRequest, reply *structs.I
&reply.QueryMeta,
state.GetQueryWatch("Services"),
func() error {
index, services, err := state.Services()
var index uint64
var services structs.Services
var err error
if len(args.NodeMetaFilters) > 0 {
index, services, err = state.ServicesByNodeMeta(args.NodeMetaFilters)
} else {
index, services, err = state.Services()
}
if err != nil {
return err
}
@ -222,6 +284,7 @@ func (c *Catalog) NodeServices(args *structs.NodeSpecificRequest, reply *structs
if err != nil {
return err
}
reply.Index, reply.NodeServices = index, services
return c.srv.filterACL(args.Token, reply)
})

618
consul/catalog_endpoint_test.go
View File

@ -14,7 +14,7 @@ import (
"github.com/hashicorp/net-rpc-msgpackrpc"
)
func TestCatalogRegister(t *testing.T) {
func TestCatalog_Register(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -30,6 +30,9 @@ func TestCatalogRegister(t *testing.T) {
Tags: []string{"master"},
Port: 8000,
},
Check: &structs.HealthCheck{
ServiceID: "db",
},
}
var out struct{}
@ -46,11 +49,12 @@ func TestCatalogRegister(t *testing.T) {
})
}
func TestCatalogRegister_ACLDeny(t *testing.T) {
func TestCatalog_Register_ACLDeny(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -59,22 +63,25 @@ func TestCatalogRegister_ACLDeny(t *testing.T) {
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Create the ACL
// Create the ACL.
arg := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: testRegisterRules,
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
service "foo" {
policy = "write"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var out string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &arg, &out); err != nil {
var id string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &arg, &id); err != nil {
t.Fatalf("err: %v", err)
}
id := out
argR := structs.RegisterRequest{
Datacenter: "dc1",
@ -89,19 +96,61 @@ func TestCatalogRegister_ACLDeny(t *testing.T) {
}
var outR struct{}
// This should fail since we are writing to the "db" service, which isn't
// allowed.
err := msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// The "foo" service should work, though.
argR.Service.Service = "foo"
err = msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR)
if err != nil {
t.Fatalf("err: %v", err)
}
// Try the special case for the "consul" service that allows it no matter
// what with pre-version 8 ACL enforcement.
argR.Service.Service = "consul"
err = msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR)
if err != nil {
t.Fatalf("err: %v", err)
}
// Make sure the exception goes away when we turn on version 8 ACL
// enforcement.
s1.config.ACLEnforceVersion8 = true
err = msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Register a db service using the root token.
argR.Service.Service = "db"
argR.Service.ID = "my-id"
argR.Token = "root"
err = msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR)
if err != nil {
t.Fatalf("err: %v", err)
}
// Prove that we are properly looking up the node services and passing
// that to the ACL helper. We can vet the helper independently in its
// own unit test after this. This is trying to register over the db
// service we created above, which is a check that depends on looking
// at the existing registration data with that service ID. This is a new
// check for version 8.
argR.Service.Service = "foo"
argR.Service.ID = "my-id"
argR.Token = id
err = msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
}
func TestCatalogRegister_ForwardLeader(t *testing.T) {
func TestCatalog_Register_ForwardLeader(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -148,7 +197,7 @@ func TestCatalogRegister_ForwardLeader(t *testing.T) {
}
}
func TestCatalogRegister_ForwardDC(t *testing.T) {
func TestCatalog_Register_ForwardDC(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -184,7 +233,7 @@ func TestCatalogRegister_ForwardDC(t *testing.T) {
}
}
func TestCatalogDeregister(t *testing.T) {
func TestCatalog_Deregister(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -209,7 +258,218 @@ func TestCatalogDeregister(t *testing.T) {
}
}
func TestCatalogListDatacenters(t *testing.T) {
func TestCatalog_Deregister_ACLDeny(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Create the ACL.
arg := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
node "node" {
policy = "write"
}
service "service" {
policy = "write"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var id string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &arg, &id); err != nil {
t.Fatalf("err: %v", err)
}
// Register a node, node check, service, and service check.
argR := structs.RegisterRequest{
Datacenter: "dc1",
Node: "node",
Address: "127.0.0.1",
Service: &structs.NodeService{
Service: "service",
Port: 8000,
},
Checks: structs.HealthChecks{
&structs.HealthCheck{
Node: "node",
CheckID: "node-check",
},
&structs.HealthCheck{
Node: "node",
CheckID: "service-check",
ServiceID: "service",
},
},
WriteRequest: structs.WriteRequest{Token: id},
}
var outR struct{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR); err != nil {
t.Fatalf("err: %v", err)
}
// First pass with version 8 ACL enforcement disabled, we should be able
// to deregister everything even without a token.
var err error
var out struct{}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "service-check"}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "node-check"}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
ServiceID: "service"}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node"}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
// Turn on version 8 ACL enforcement and put the catalog entry back.
s1.config.ACLEnforceVersion8 = true
if err := msgpackrpc.CallWithCodec(codec, "Catalog.Register", &argR, &outR); err != nil {
t.Fatalf("err: %v", err)
}
// Second pass with version 8 ACL enforcement enabled, these should all
// get rejected.
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "service-check"}, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "node-check"}, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
ServiceID: "service"}, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node"}, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Third pass these should all go through with the token set.
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "service-check",
WriteRequest: structs.WriteRequest{
Token: id,
}}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "node-check",
WriteRequest: structs.WriteRequest{
Token: id,
}}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
ServiceID: "service",
WriteRequest: structs.WriteRequest{
Token: id,
}}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
WriteRequest: structs.WriteRequest{
Token: id,
}}, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
// Try a few error cases.
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
ServiceID: "nope",
WriteRequest: structs.WriteRequest{
Token: id,
}}, &out)
if err == nil || !strings.Contains(err.Error(), "Unknown service") {
t.Fatalf("err: %v", err)
}
err = msgpackrpc.CallWithCodec(codec, "Catalog.Deregister",
&structs.DeregisterRequest{
Datacenter: "dc1",
Node: "node",
CheckID: "nope",
WriteRequest: structs.WriteRequest{
Token: id,
}}, &out)
if err == nil || !strings.Contains(err.Error(), "Unknown check") {
t.Fatalf("err: %v", err)
}
}
func TestCatalog_ListDatacenters(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -246,7 +506,7 @@ func TestCatalogListDatacenters(t *testing.T) {
}
}
func TestCatalogListDatacenters_DistanceSort(t *testing.T) {
func TestCatalog_ListDatacenters_DistanceSort(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -290,7 +550,7 @@ func TestCatalogListDatacenters_DistanceSort(t *testing.T) {
}
}
func TestCatalogListNodes(t *testing.T) {
func TestCatalog_ListNodes(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -332,7 +592,71 @@ func TestCatalogListNodes(t *testing.T) {
}
}
func TestCatalogListNodes_StaleRaad(t *testing.T) {
func TestCatalog_ListNodes_MetaFilter(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Add a new node with the right meta k/v pair
node := &structs.Node{Node: "foo", Address: "127.0.0.1", Meta: map[string]string{"somekey": "somevalue"}}
if err := s1.fsm.State().EnsureNode(1, node); err != nil {
t.Fatalf("err: %v", err)
}
// Filter by a specific meta k/v pair
args := structs.DCSpecificRequest{
Datacenter: "dc1",
NodeMetaFilters: map[string]string{
"somekey": "somevalue",
},
}
var out structs.IndexedNodes
testutil.WaitForResult(func() (bool, error) {
msgpackrpc.CallWithCodec(codec, "Catalog.ListNodes", &args, &out)
return len(out.Nodes) == 1, nil
}, func(err error) {
t.Fatalf("err: %v", err)
})
// Verify that only the correct node was returned
if out.Nodes[0].Node != "foo" {
t.Fatalf("bad: %v", out)
}
if out.Nodes[0].Address != "127.0.0.1" {
t.Fatalf("bad: %v", out)
}
if v, ok := out.Nodes[0].Meta["somekey"]; !ok || v != "somevalue" {
t.Fatalf("bad: %v", out)
}
// Now filter on a nonexistent meta k/v pair
args = structs.DCSpecificRequest{
Datacenter: "dc1",
NodeMetaFilters: map[string]string{
"somekey": "invalid",
},
}
out = structs.IndexedNodes{}
err := msgpackrpc.CallWithCodec(codec, "Catalog.ListNodes", &args, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
// Should get an empty list of nodes back
testutil.WaitForResult(func() (bool, error) {
msgpackrpc.CallWithCodec(codec, "Catalog.ListNodes", &args, &out)
return len(out.Nodes) == 0, nil
}, func(err error) {
t.Fatalf("err: %v", err)
})
}
func TestCatalog_ListNodes_StaleRaad(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -400,7 +724,7 @@ func TestCatalogListNodes_StaleRaad(t *testing.T) {
}
}
func TestCatalogListNodes_ConsistentRead_Fail(t *testing.T) {
func TestCatalog_ListNodes_ConsistentRead_Fail(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -450,7 +774,7 @@ func TestCatalogListNodes_ConsistentRead_Fail(t *testing.T) {
}
}
func TestCatalogListNodes_ConsistentRead(t *testing.T) {
func TestCatalog_ListNodes_ConsistentRead(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -498,7 +822,7 @@ func TestCatalogListNodes_ConsistentRead(t *testing.T) {
}
}
func TestCatalogListNodes_DistanceSort(t *testing.T) {
func TestCatalog_ListNodes_DistanceSort(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -586,7 +910,84 @@ func TestCatalogListNodes_DistanceSort(t *testing.T) {
}
}
func BenchmarkCatalogListNodes(t *testing.B) {
func TestCatalog_ListNodes_ACLFilter(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// We scope the reply in each of these since msgpack won't clear out an
// existing slice if the incoming one is nil, so it's best to start
// clean each time.
// Prior to version 8, the node policy should be ignored.
args := structs.DCSpecificRequest{
Datacenter: "dc1",
}
{
reply := structs.IndexedNodes{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.ListNodes", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if len(reply.Nodes) != 1 {
t.Fatalf("bad: %v", reply.Nodes)
}
}
// Now turn on version 8 enforcement and try again.
s1.config.ACLEnforceVersion8 = true
{
reply := structs.IndexedNodes{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.ListNodes", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if len(reply.Nodes) != 0 {
t.Fatalf("bad: %v", reply.Nodes)
}
}
// Create an ACL that can read the node.
arg := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: fmt.Sprintf(`
node "%s" {
policy = "read"
}
`, s1.config.NodeName),
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var id string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &arg, &id); err != nil {
t.Fatalf("err: %v", err)
}
// Now try with the token and it will go through.
args.Token = id
{
reply := structs.IndexedNodes{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.ListNodes", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if len(reply.Nodes) != 1 {
t.Fatalf("bad: %v", reply.Nodes)
}
}
}
func Benchmark_Catalog_ListNodes(t *testing.B) {
dir1, s1 := testServer(nil)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -609,7 +1010,7 @@ func BenchmarkCatalogListNodes(t *testing.B) {
}
}
func TestCatalogListServices(t *testing.T) {
func TestCatalog_ListServices(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -659,7 +1060,70 @@ func TestCatalogListServices(t *testing.T) {
}
}
func TestCatalogListServices_Blocking(t *testing.T) {
func TestCatalog_ListServices_MetaFilter(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Add a new node with the right meta k/v pair
node := &structs.Node{Node: "foo", Address: "127.0.0.1", Meta: map[string]string{"somekey": "somevalue"}}
if err := s1.fsm.State().EnsureNode(1, node); err != nil {
t.Fatalf("err: %v", err)
}
// Add a service to the new node
if err := s1.fsm.State().EnsureService(2, "foo", &structs.NodeService{ID: "db", Service: "db", Tags: []string{"primary"}, Address: "127.0.0.1", Port: 5000}); err != nil {
t.Fatalf("err: %v", err)
}
// Filter by a specific meta k/v pair
args := structs.DCSpecificRequest{
Datacenter: "dc1",
NodeMetaFilters: map[string]string{
"somekey": "somevalue",
},
}
var out structs.IndexedServices
if err := msgpackrpc.CallWithCodec(codec, "Catalog.ListServices", &args, &out); err != nil {
t.Fatalf("err: %v", err)
}
if len(out.Services) != 1 {
t.Fatalf("bad: %v", out)
}
if out.Services["db"] == nil {
t.Fatalf("bad: %v", out.Services["db"])
}
if len(out.Services["db"]) != 1 {
t.Fatalf("bad: %v", out)
}
if out.Services["db"][0] != "primary" {
t.Fatalf("bad: %v", out)
}
// Now filter on a nonexistent meta k/v pair
args = structs.DCSpecificRequest{
Datacenter: "dc1",
NodeMetaFilters: map[string]string{
"somekey": "invalid",
},
}
out = structs.IndexedServices{}
err := msgpackrpc.CallWithCodec(codec, "Catalog.ListServices", &args, &out)
if err != nil {
t.Fatalf("err: %v", err)
}
// Should get an empty list of nodes back
if len(out.Services) != 0 {
t.Fatalf("bad: %v", out.Services)
}
}
func TestCatalog_ListServices_Blocking(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -717,7 +1181,7 @@ func TestCatalogListServices_Blocking(t *testing.T) {
}
}
func TestCatalogListServices_Timeout(t *testing.T) {
func TestCatalog_ListServices_Timeout(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -758,7 +1222,7 @@ func TestCatalogListServices_Timeout(t *testing.T) {
}
}
func TestCatalogListServices_Stale(t *testing.T) {
func TestCatalog_ListServices_Stale(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -795,7 +1259,7 @@ func TestCatalogListServices_Stale(t *testing.T) {
}
}
func TestCatalogListServiceNodes(t *testing.T) {
func TestCatalog_ListServiceNodes(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -844,7 +1308,7 @@ func TestCatalogListServiceNodes(t *testing.T) {
}
}
func TestCatalogListServiceNodes_DistanceSort(t *testing.T) {
func TestCatalog_ListServiceNodes_DistanceSort(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -931,7 +1395,7 @@ func TestCatalogListServiceNodes_DistanceSort(t *testing.T) {
}
}
func TestCatalogNodeServices(t *testing.T) {
func TestCatalog_NodeServices(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -981,7 +1445,7 @@ func TestCatalogNodeServices(t *testing.T) {
}
// Used to check for a regression against a known bug
func TestCatalogRegister_FailedCase1(t *testing.T) {
func TestCatalog_Register_FailedCase1(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -1032,6 +1496,7 @@ func testACLFilterServer(t *testing.T) (dir, token string, srv *Server, codec rp
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
codec = rpcClient(t, srv)
@ -1042,9 +1507,13 @@ func testACLFilterServer(t *testing.T) (dir, token string, srv *Server, codec rp
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: testRegisterRules,
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
service "foo" {
policy = "write"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
@ -1158,6 +1627,87 @@ func TestCatalog_ServiceNodes_FilterACL(t *testing.T) {
t.Fatalf("bad: %#v", reply.ServiceNodes)
}
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}
func TestCatalog_NodeServices_ACLDeny(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Prior to version 8, the node policy should be ignored.
args := structs.NodeSpecificRequest{
Datacenter: "dc1",
Node: s1.config.NodeName,
}
reply := structs.IndexedNodeServices{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.NodeServices", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if reply.NodeServices == nil {
t.Fatalf("should not be nil")
}
// Now turn on version 8 enforcement and try again.
s1.config.ACLEnforceVersion8 = true
if err := msgpackrpc.CallWithCodec(codec, "Catalog.NodeServices", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if reply.NodeServices != nil {
t.Fatalf("should not nil")
}
// Create an ACL that can read the node.
arg := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: fmt.Sprintf(`
node "%s" {
policy = "read"
}
`, s1.config.NodeName),
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var id string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &arg, &id); err != nil {
t.Fatalf("err: %v", err)
}
// Now try with the token and it will go through.
args.Token = id
if err := msgpackrpc.CallWithCodec(codec, "Catalog.NodeServices", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if reply.NodeServices == nil {
t.Fatalf("should not be nil")
}
// Make sure an unknown node doesn't cause trouble.
args.Node = "nope"
if err := msgpackrpc.CallWithCodec(codec, "Catalog.NodeServices", &args, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if reply.NodeServices != nil {
t.Fatalf("should not nil")
}
}
func TestCatalog_NodeServices_FilterACL(t *testing.T) {
@ -1189,9 +1739,3 @@ func TestCatalog_NodeServices_FilterACL(t *testing.T) {
t.Fatalf("bad: %#v", reply.NodeServices)
}
}
var testRegisterRules = `
service "foo" {
policy = "write"
}
`

25
consul/config.go
View File

@ -155,6 +155,11 @@ type Config struct {
// backwards compatibility as well.
ACLToken string
// ACLAgentToken is the default token used to make requests for the agent
// itself, such as for registering itself with the catalog. If not
// configured, the ACLToken will be used.
ACLAgentToken string
// ACLMasterToken is used to bootstrap the ACL system. It should be specified
// on the servers in the ACLDatacenter. When the leader comes online, it ensures
// that the Master token is available. This provides the initial token.
@ -200,6 +205,10 @@ type Config struct {
// used to limit the amount of Raft bandwidth used for replication.
ACLReplicationApplyLimit int
// ACLEnforceVersion8 is used to gate a set of ACL policy features that
// are opt-in prior to Consul 0.8 and opt-out in Consul 0.8 and later.
ACLEnforceVersion8 bool
// TombstoneTTL is used to control how long KV tombstones are retained.
// This provides a window of time where the X-Consul-Index is monotonic.
// Outside this window, the index may not be monotonic. This is a result
@ -350,6 +359,9 @@ func DefaultConfig() *Config {
// Disable shutdown on removal
conf.RaftConfig.ShutdownOnRemove = false
// Check every 5 seconds to see if there are enough new entries for a snapshot
conf.RaftConfig.SnapshotInterval = 5 * time.Second
return conf
}
@ -366,6 +378,7 @@ func (c *Config) ScaleRaft(raftMultRaw uint) {
c.RaftConfig.LeaderLeaseTimeout = raftMult * def.LeaderLeaseTimeout
}
// tlsConfig maps this config into a tlsutil config.
func (c *Config) tlsConfig() *tlsutil.Config {
tlsConf := &tlsutil.Config{
VerifyIncoming: c.VerifyIncoming,
@ -380,3 +393,15 @@ func (c *Config) tlsConfig() *tlsutil.Config {
}
return tlsConf
}
// GetTokenForAgent returns the token the agent should use for its own internal
// operations, such as registering itself with the catalog.
func (c *Config) GetTokenForAgent() string {
if c.ACLAgentToken != "" {
return c.ACLAgentToken
} else if c.ACLToken != "" {
return c.ACLToken
} else {
return ""
}
}

20
consul/config_test.go Normal file
View File

@ -0,0 +1,20 @@
package consul
import (
"testing"
)
func TestConfig_GetTokenForAgent(t *testing.T) {
config := DefaultConfig()
if token := config.GetTokenForAgent(); token != "" {
t.Fatalf("bad: %s", token)
}
config.ACLToken = "hello"
if token := config.GetTokenForAgent(); token != "hello" {
t.Fatalf("bad: %s", token)
}
config.ACLAgentToken = "world"
if token := config.GetTokenForAgent(); token != "world" {
t.Fatalf("bad: %s", token)
}
}

14
consul/coordinate_endpoint.go
View File

@ -119,6 +119,17 @@ func (c *Coordinate) Update(args *structs.CoordinateUpdateRequest, reply *struct
return fmt.Errorf("rejected bad coordinate: %v", args.Coord)
}
// Fetch the ACL token, if any, and enforce the node policy if enabled.
acl, err := c.srv.resolveToken(args.Token)
if err != nil {
return err
}
if acl != nil && c.srv.config.ACLEnforceVersion8 {
if !acl.NodeWrite(args.Node) {
return permissionDeniedErr
}
}
// Add the coordinate to the map of pending updates.
c.updatesLock.Lock()
c.updates[args.Node] = args.Coord
@ -173,6 +184,9 @@ func (c *Coordinate) ListNodes(args *structs.DCSpecificRequest, reply *structs.I
}
reply.Index, reply.Coordinates = index, coords
if err := c.srv.filterACL(args.Token, reply); err != nil {
return err
}
return nil
})
}

238
consul/coordinate_endpoint_test.go
View File

@ -187,6 +187,87 @@ func TestCoordinate_Update(t *testing.T) {
}
}
func TestCoordinate_Update_ACLDeny(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Register some nodes.
nodes := []string{"node1", "node2"}
for _, node := range nodes {
req := structs.RegisterRequest{
Datacenter: "dc1",
Node: node,
Address: "127.0.0.1",
}
var reply struct{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.Register", &req, &reply); err != nil {
t.Fatalf("err: %v", err)
}
}
// Send an update for the first node. This should go through since we
// don't have version 8 ACLs enforced yet.
req := structs.CoordinateUpdateRequest{
Datacenter: "dc1",
Node: "node1",
Coord: generateRandomCoordinate(),
}
var out struct{}
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &req, &out); err != nil {
t.Fatalf("err: %v", err)
}
// Now turn on version 8 enforcement and try again.
s1.config.ACLEnforceVersion8 = true
err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &req, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Create an ACL that can write to the node.
arg := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
node "node1" {
policy = "write"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var id string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &arg, &id); err != nil {
t.Fatalf("err: %v", err)
}
// With the token, it should now go through.
req.Token = id
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &req, &out); err != nil {
t.Fatalf("err: %v", err)
}
// But it should be blocked for the other node.
req.Node = "node2"
err = msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &req, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
}
func TestCoordinate_ListDatacenters(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
@ -240,8 +321,7 @@ func TestCoordinate_ListNodes(t *testing.T) {
}
}
// Send coordinate updates for a few nodes, waiting a little while for
// the batch update to run.
// Send coordinate updates for a few nodes.
arg1 := structs.CoordinateUpdateRequest{
Datacenter: "dc1",
Node: "foo",
@ -269,9 +349,120 @@ func TestCoordinate_ListNodes(t *testing.T) {
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &arg3, &out); err != nil {
t.Fatalf("err: %v", err)
}
time.Sleep(3 * s1.config.CoordinateUpdatePeriod)
// Now query back for all the nodes.
testutil.WaitForResult(func() (bool, error) {
arg := structs.DCSpecificRequest{
Datacenter: "dc1",
}
resp := structs.IndexedCoordinates{}
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.ListNodes", &arg, &resp); err != nil {
t.Fatalf("err: %v", err)
}
if len(resp.Coordinates) != 3 ||
resp.Coordinates[0].Node != "bar" ||
resp.Coordinates[1].Node != "baz" ||
resp.Coordinates[2].Node != "foo" {
return false, fmt.Errorf("bad: %v", resp.Coordinates)
}
verifyCoordinatesEqual(t, resp.Coordinates[0].Coord, arg2.Coord) // bar
verifyCoordinatesEqual(t, resp.Coordinates[1].Coord, arg3.Coord) // baz
verifyCoordinatesEqual(t, resp.Coordinates[2].Coord, arg1.Coord) // foo
return true, nil
}, func(err error) { t.Fatalf("err: %v", err) })
}
func TestCoordinate_ListNodes_ACLFilter(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Register some nodes.
nodes := []string{"foo", "bar", "baz"}
for _, node := range nodes {
req := structs.RegisterRequest{
Datacenter: "dc1",
Node: node,
Address: "127.0.0.1",
WriteRequest: structs.WriteRequest{
Token: "root",
},
}
var reply struct{}
if err := msgpackrpc.CallWithCodec(codec, "Catalog.Register", &req, &reply); err != nil {
t.Fatalf("err: %v", err)
}
}
// Send coordinate updates for a few nodes.
arg1 := structs.CoordinateUpdateRequest{
Datacenter: "dc1",
Node: "foo",
Coord: generateRandomCoordinate(),
WriteRequest: structs.WriteRequest{
Token: "root",
},
}
var out struct{}
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &arg1, &out); err != nil {
t.Fatalf("err: %v", err)
}
arg2 := structs.CoordinateUpdateRequest{
Datacenter: "dc1",
Node: "bar",
Coord: generateRandomCoordinate(),
WriteRequest: structs.WriteRequest{
Token: "root",
},
}
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &arg2, &out); err != nil {
t.Fatalf("err: %v", err)
}
arg3 := structs.CoordinateUpdateRequest{
Datacenter: "dc1",
Node: "baz",
Coord: generateRandomCoordinate(),
WriteRequest: structs.WriteRequest{
Token: "root",
},
}
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.Update", &arg3, &out); err != nil {
t.Fatalf("err: %v", err)
}
// Wait for all the coordinate updates to apply. Since we aren't
// enforcing version 8 ACLs, this should also allow us to read
// everything back without a token.
testutil.WaitForResult(func() (bool, error) {
arg := structs.DCSpecificRequest{
Datacenter: "dc1",
}
resp := structs.IndexedCoordinates{}
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.ListNodes", &arg, &resp); err != nil {
t.Fatalf("err: %v", err)
}
if len(resp.Coordinates) == 3 {
return true, nil
}
return false, fmt.Errorf("bad: %v", resp.Coordinates)
}, func(err error) { t.Fatalf("err: %v", err) })
// Now that we've waited for the batch processing to ingest the
// coordinates we can do the rest of the requests without the loop. We
// will start by turning on version 8 ACL support which should block
// everything.
s1.config.ACLEnforceVersion8 = true
arg := structs.DCSpecificRequest{
Datacenter: "dc1",
}
@ -279,13 +470,38 @@ func TestCoordinate_ListNodes(t *testing.T) {
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.ListNodes", &arg, &resp); err != nil {
t.Fatalf("err: %v", err)
}
if len(resp.Coordinates) != 3 ||
resp.Coordinates[0].Node != "bar" ||
resp.Coordinates[1].Node != "baz" ||
resp.Coordinates[2].Node != "foo" {
t.Fatalf("bad: %v", resp.Coordinates)
if len(resp.Coordinates) != 0 {
t.Fatalf("bad: %#v", resp.Coordinates)
}
// Create an ACL that can read one of the nodes.
var id string
{
req := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
node "foo" {
policy = "read"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &req, &id); err != nil {
t.Fatalf("err: %v", err)
}
}
// With the token, it should now go through.
arg.Token = id
if err := msgpackrpc.CallWithCodec(codec, "Coordinate.ListNodes", &arg, &resp); err != nil {
t.Fatalf("err: %v", err)
}
if len(resp.Coordinates) != 1 || resp.Coordinates[0].Node != "foo" {
t.Fatalf("bad: %#v", resp.Coordinates)
}
verifyCoordinatesEqual(t, resp.Coordinates[0].Coord, arg2.Coord) // bar
verifyCoordinatesEqual(t, resp.Coordinates[1].Coord, arg3.Coord) // baz
verifyCoordinatesEqual(t, resp.Coordinates[2].Coord, arg1.Coord) // foo
}

4
consul/fsm.go
View File

@ -127,7 +127,9 @@ func (c *consulFSM) applyDeregister(buf []byte, index uint64) interface{} {
panic(fmt.Errorf("failed to decode request: %v", err))
}
// Either remove the service entry or the whole node
// Either remove the service entry or the whole node. The precedence
// here is also baked into vetDeregisterWithACL() in acl.go, so if you
// make changes here, be sure to also adjust the code over there.
if req.ServiceID != "" {
if err := c.state.DeleteService(index, req.Node, req.ServiceID); err != nil {
c.logger.Printf("[INFO] consul.fsm: DeleteNodeService failed: %v", err)

24
consul/health_endpoint_test.go
View File

@ -507,6 +507,12 @@ func TestHealth_NodeChecks_FilterACL(t *testing.T) {
if !found {
t.Fatalf("bad: %#v", reply.HealthChecks)
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}
func TestHealth_ServiceChecks_FilterACL(t *testing.T) {
@ -543,6 +549,12 @@ func TestHealth_ServiceChecks_FilterACL(t *testing.T) {
if len(reply.HealthChecks) != 0 {
t.Fatalf("bad: %#v", reply.HealthChecks)
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}
func TestHealth_ServiceNodes_FilterACL(t *testing.T) {
@ -572,6 +584,12 @@ func TestHealth_ServiceNodes_FilterACL(t *testing.T) {
if len(reply.Nodes) != 0 {
t.Fatalf("bad: %#v", reply.Nodes)
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}
func TestHealth_ChecksInState_FilterACL(t *testing.T) {
@ -602,4 +620,10 @@ func TestHealth_ChecksInState_FilterACL(t *testing.T) {
if !found {
t.Fatalf("missing service 'foo': %#v", reply.HealthChecks)
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}

12
consul/internal_endpoint_test.go
View File

@ -284,6 +284,12 @@ func TestInternal_NodeInfo_FilterACL(t *testing.T) {
t.Fatalf("bad: %#v", info.Services)
}
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}
func TestInternal_NodeDump_FilterACL(t *testing.T) {
@ -327,6 +333,12 @@ func TestInternal_NodeDump_FilterACL(t *testing.T) {
t.Fatalf("bad: %#v", info.Services)
}
}
// We've already proven that we call the ACL filtering function so we
// test node filtering down in acl.go for node cases. This also proves
// that we respect the version 8 ACL flag, since the test server sets
// that to false (the regression value of *not* changing this is better
// for now until we change the sense of the version 8 ACL flag).
}
func TestInternal_EventFire_Token(t *testing.T) {

6
consul/leader.go
View File

@ -428,7 +428,7 @@ AFTER_CHECK:
Status: structs.HealthPassing,
Output: SerfCheckAliveOutput,
},
WriteRequest: structs.WriteRequest{Token: s.config.ACLToken},
WriteRequest: structs.WriteRequest{Token: s.config.GetTokenForAgent()},
}
var out struct{}
return s.endpoints.Catalog.Register(&req, &out)
@ -469,7 +469,7 @@ func (s *Server) handleFailedMember(member serf.Member) error {
Status: structs.HealthCritical,
Output: SerfCheckFailedOutput,
},
WriteRequest: structs.WriteRequest{Token: s.config.ACLToken},
WriteRequest: structs.WriteRequest{Token: s.config.GetTokenForAgent()},
}
var out struct{}
return s.endpoints.Catalog.Register(&req, &out)
@ -612,7 +612,7 @@ func (s *Server) reapTombstones(index uint64) {
Datacenter: s.config.Datacenter,
Op: structs.TombstoneReap,
ReapIndex: index,
WriteRequest: structs.WriteRequest{Token: s.config.ACLToken},
WriteRequest: structs.WriteRequest{Token: s.config.GetTokenForAgent()},
}
_, err := s.raftApply(structs.TombstoneRequestType, &req)
if err != nil {

12
consul/prepared_query_endpoint.go
View File

@ -96,7 +96,7 @@ func (p *PreparedQuery) Apply(args *structs.PreparedQueryRequest, reply *string)
// Parse the query and prep it for the state store.
switch args.Op {
case structs.PreparedQueryCreate, structs.PreparedQueryUpdate:
if err := parseQuery(args.Query); err != nil {
if err := parseQuery(args.Query, p.srv.config.ACLEnforceVersion8); err != nil {
return fmt.Errorf("Invalid prepared query: %v", err)
}
@ -125,7 +125,7 @@ func (p *PreparedQuery) Apply(args *structs.PreparedQueryRequest, reply *string)
// update operation. Some of the fields are not checked or are partially
// checked, as noted in the comments below. This also updates all the parsed
// fields of the query.
func parseQuery(query *structs.PreparedQuery) error {
func parseQuery(query *structs.PreparedQuery, enforceVersion8 bool) error {
// We skip a few fields:
// - ID is checked outside this fn.
// - Name is optional with no restrictions, except for uniqueness which
@ -133,10 +133,16 @@ func parseQuery(query *structs.PreparedQuery) error {
// names do not overlap with IDs, which is also checked during the
// transaction. Otherwise, people could "steal" queries that they don't
// have proper ACL rights to change.
// - Session is optional and checked for integrity during the transaction.
// - Template is checked during the transaction since that's where we
// compile it.
// Anonymous queries require a session or need to be part of a template.
if enforceVersion8 {
if query.Name == "" && query.Template.Type == "" && query.Session == "" {
return fmt.Errorf("Must be bound to a session")
}
}
// Token is checked when the query is executed, but we do make sure the
// user hasn't accidentally pasted-in the special redacted token name,
// which if we allowed in would be super hard to debug and understand.

182
consul/prepared_query_endpoint_test.go
View File

@ -32,6 +32,7 @@ func TestPreparedQuery_Apply(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "redis",
},
@ -515,6 +516,7 @@ func TestPreparedQuery_Apply_ForwardLeader(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "redis",
},
@ -531,53 +533,77 @@ func TestPreparedQuery_Apply_ForwardLeader(t *testing.T) {
func TestPreparedQuery_parseQuery(t *testing.T) {
query := &structs.PreparedQuery{}
err := parseQuery(query)
err := parseQuery(query, true)
if err == nil || !strings.Contains(err.Error(), "Must be bound to a session") {
t.Fatalf("bad: %v", err)
}
query.Session = "adf4238a-882b-9ddc-4a9d-5b6758e4159e"
err = parseQuery(query, true)
if err == nil || !strings.Contains(err.Error(), "Must provide a Service") {
t.Fatalf("bad: %v", err)
}
query.Service.Service = "foo"
if err := parseQuery(query); err != nil {
t.Fatalf("err: %v", err)
}
query.Token = redactedToken
err = parseQuery(query)
if err == nil || !strings.Contains(err.Error(), "Bad Token") {
query.Session = ""
query.Template.Type = "some-kind-of-template"
err = parseQuery(query, true)
if err == nil || !strings.Contains(err.Error(), "Must provide a Service") {
t.Fatalf("bad: %v", err)
}
query.Token = "adf4238a-882b-9ddc-4a9d-5b6758e4159e"
if err := parseQuery(query); err != nil {
t.Fatalf("err: %v", err)
}
query.Service.Failover.NearestN = -1
err = parseQuery(query)
if err == nil || !strings.Contains(err.Error(), "Bad NearestN") {
query.Template.Type = ""
err = parseQuery(query, false)
if err == nil || !strings.Contains(err.Error(), "Must provide a Service") {
t.Fatalf("bad: %v", err)
}
query.Service.Failover.NearestN = 3
if err := parseQuery(query); err != nil {
t.Fatalf("err: %v", err)
}
// None of the rest of these care about version 8 ACL enforcement.
for _, version8 := range []bool{true, false} {
query = &structs.PreparedQuery{}
query.Session = "adf4238a-882b-9ddc-4a9d-5b6758e4159e"
query.Service.Service = "foo"
if err := parseQuery(query, version8); err != nil {
t.Fatalf("err: %v", err)
}
query.DNS.TTL = "two fortnights"
err = parseQuery(query)
if err == nil || !strings.Contains(err.Error(), "Bad DNS TTL") {
t.Fatalf("bad: %v", err)
}
query.Token = redactedToken
err = parseQuery(query, version8)
if err == nil || !strings.Contains(err.Error(), "Bad Token") {
t.Fatalf("bad: %v", err)
}
query.DNS.TTL = "-3s"
err = parseQuery(query)
if err == nil || !strings.Contains(err.Error(), "must be >=0") {
t.Fatalf("bad: %v", err)
}
query.Token = "adf4238a-882b-9ddc-4a9d-5b6758e4159e"
if err := parseQuery(query, version8); err != nil {
t.Fatalf("err: %v", err)
}
query.DNS.TTL = "3s"
if err := parseQuery(query); err != nil {
t.Fatalf("err: %v", err)
query.Service.Failover.NearestN = -1
err = parseQuery(query, version8)
if err == nil || !strings.Contains(err.Error(), "Bad NearestN") {
t.Fatalf("bad: %v", err)
}
query.Service.Failover.NearestN = 3
if err := parseQuery(query, version8); err != nil {
t.Fatalf("err: %v", err)
}
query.DNS.TTL = "two fortnights"
err = parseQuery(query, version8)
if err == nil || !strings.Contains(err.Error(), "Bad DNS TTL") {
t.Fatalf("bad: %v", err)
}
query.DNS.TTL = "-3s"
err = parseQuery(query, version8)
if err == nil || !strings.Contains(err.Error(), "must be >=0") {
t.Fatalf("bad: %v", err)
}
query.DNS.TTL = "3s"
if err := parseQuery(query, version8); err != nil {
t.Fatalf("err: %v", err)
}
}
}
@ -917,9 +943,25 @@ func TestPreparedQuery_Get(t *testing.T) {
}
}
// Create a session.
var session string
{
req := structs.SessionRequest{
Datacenter: "dc1",
Op: structs.SessionCreate,
Session: structs.Session{
Node: s1.config.NodeName,
},
}
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &req, &session); err != nil {
t.Fatalf("err: %v", err)
}
}
// Now update the query to take away its name.
query.Op = structs.PreparedQueryUpdate
query.Query.Name = ""
query.Query.Session = session
if err := msgpackrpc.CallWithCodec(codec, "PreparedQuery.Apply", &query, &reply); err != nil {
t.Fatalf("err: %v", err)
}
@ -1170,9 +1212,25 @@ func TestPreparedQuery_List(t *testing.T) {
}
}
// Create a session.
var session string
{
req := structs.SessionRequest{
Datacenter: "dc1",
Op: structs.SessionCreate,
Session: structs.Session{
Node: s1.config.NodeName,
},
}
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &req, &session); err != nil {
t.Fatalf("err: %v", err)
}
}
// Now take away the query name.
query.Op = structs.PreparedQueryUpdate
query.Query.Name = ""
query.Query.Session = session
if err := msgpackrpc.CallWithCodec(codec, "PreparedQuery.Apply", &query, &reply); err != nil {
t.Fatalf("err: %v", err)
}
@ -1359,6 +1417,7 @@ func TestPreparedQuery_Execute(t *testing.T) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -1451,6 +1510,7 @@ func TestPreparedQuery_Execute(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "foo",
},
@ -2079,6 +2139,58 @@ func TestPreparedQuery_Execute(t *testing.T) {
}
}
// Turn on version 8 ACLs, which will start to filter even with the exec
// token.
s1.config.ACLEnforceVersion8 = true
{
req := structs.PreparedQueryExecuteRequest{
Datacenter: "dc1",
QueryIDOrName: query.Query.ID,
QueryOptions: structs.QueryOptions{Token: execToken},
}
var reply structs.PreparedQueryExecuteResponse
if err := msgpackrpc.CallWithCodec(codec1, "PreparedQuery.Execute", &req, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if len(reply.Nodes) != 0 ||
reply.Datacenter != "dc1" || reply.Failovers != 0 ||
reply.Service != query.Query.Service.Service ||
!reflect.DeepEqual(reply.DNS, query.Query.DNS) ||
!reply.QueryMeta.KnownLeader {
t.Fatalf("bad: %v", reply)
}
}
// Revert version 8 ACLs and make sure the query works again.
s1.config.ACLEnforceVersion8 = false
{
req := structs.PreparedQueryExecuteRequest{
Datacenter: "dc1",
QueryIDOrName: query.Query.ID,
QueryOptions: structs.QueryOptions{Token: execToken},
}
var reply structs.PreparedQueryExecuteResponse
if err := msgpackrpc.CallWithCodec(codec1, "PreparedQuery.Execute", &req, &reply); err != nil {
t.Fatalf("err: %v", err)
}
if len(reply.Nodes) != 8 ||
reply.Datacenter != "dc1" || reply.Failovers != 0 ||
reply.Service != query.Query.Service.Service ||
!reflect.DeepEqual(reply.DNS, query.Query.DNS) ||
!reply.QueryMeta.KnownLeader {
t.Fatalf("bad: %v", reply)
}
for _, node := range reply.Nodes {
if node.Node.Node == "node1" || node.Node.Node == "node3" {
t.Fatalf("bad: %v", node)
}
}
}
// Now fail everything in dc1 and we should get an empty list back.
for i := 0; i < 10; i++ {
setHealth(fmt.Sprintf("node%d", i+1), structs.HealthCritical)
@ -2314,6 +2426,7 @@ func TestPreparedQuery_Execute_ForwardLeader(t *testing.T) {
Datacenter: "dc1",
Op: structs.PreparedQueryCreate,
Query: &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Service: "redis",
},
@ -2577,6 +2690,7 @@ func (m *mockQueryServer) ForwardDC(method, dc string, args interface{}, reply i
func TestPreparedQuery_queryFailover(t *testing.T) {
query := &structs.PreparedQuery{
Name: "test",
Service: structs.ServiceQuery{
Failover: structs.QueryDatacenterOptions{
NearestN: 0,

64
consul/session_endpoint.go
View File

@ -30,6 +30,33 @@ func (s *Session) Apply(args *structs.SessionRequest, reply *string) error {
return fmt.Errorf("Must provide Node")
}
// Fetch the ACL token, if any, and apply the policy.
acl, err := s.srv.resolveToken(args.Token)
if err != nil {
return err
}
if acl != nil && s.srv.config.ACLEnforceVersion8 {
switch args.Op {
case structs.SessionDestroy:
state := s.srv.fsm.State()
_, existing, err := state.SessionGet(args.Session.ID)
if err != nil {
return fmt.Errorf("Unknown session %q", args.Session.ID)
}
if !acl.SessionWrite(existing.Node) {
return permissionDeniedErr
}
case structs.SessionCreate:
if !acl.SessionWrite(args.Session.Node) {
return permissionDeniedErr
}
default:
return fmt.Errorf("Invalid session operation %q", args.Op)
}
}
// Ensure that the specified behavior is allowed
switch args.Session.Behavior {
case "":
@ -130,6 +157,9 @@ func (s *Session) Get(args *structs.SessionSpecificRequest,
} else {
reply.Sessions = nil
}
if err := s.srv.filterACL(args.Token, reply); err != nil {
return err
}
return nil
})
}
@ -154,6 +184,9 @@ func (s *Session) List(args *structs.DCSpecificRequest,
}
reply.Index, reply.Sessions = index, sessions
if err := s.srv.filterACL(args.Token, reply); err != nil {
return err
}
return nil
})
}
@ -178,6 +211,9 @@ func (s *Session) NodeSessions(args *structs.NodeSpecificRequest,
}
reply.Index, reply.Sessions = index, sessions
if err := s.srv.filterACL(args.Token, reply); err != nil {
return err
}
return nil
})
}
@ -190,21 +226,35 @@ func (s *Session) Renew(args *structs.SessionSpecificRequest,
}
defer metrics.MeasureSince([]string{"consul", "session", "renew"}, time.Now())
// Get the session, from local state
// Get the session, from local state.
state := s.srv.fsm.State()
index, session, err := state.SessionGet(args.Session)
if err != nil {
return err
}
// Reset the session TTL timer
reply.Index = index
if session != nil {
reply.Sessions = structs.Sessions{session}
if err := s.srv.resetSessionTimer(args.Session, session); err != nil {
s.srv.logger.Printf("[ERR] consul.session: Session renew failed: %v", err)
return err
if session == nil {
return nil
}
// Fetch the ACL token, if any, and apply the policy.
acl, err := s.srv.resolveToken(args.Token)
if err != nil {
return err
}
if acl != nil && s.srv.config.ACLEnforceVersion8 {
if !acl.SessionWrite(session.Node) {
return permissionDeniedErr
}
}
// Reset the session TTL timer.
reply.Sessions = structs.Sessions{session}
if err := s.srv.resetSessionTimer(args.Session, session); err != nil {
s.srv.logger.Printf("[ERR] consul.session: Session renew failed: %v", err)
return err
}
return nil
}

357
consul/session_endpoint_test.go
View File

@ -2,6 +2,7 @@ package consul
import (
"os"
"strings"
"testing"
"time"
@ -11,7 +12,7 @@ import (
"github.com/hashicorp/net-rpc-msgpackrpc"
)
func TestSessionEndpoint_Apply(t *testing.T) {
func TestSession_Apply(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -70,7 +71,7 @@ func TestSessionEndpoint_Apply(t *testing.T) {
}
}
func TestSessionEndpoint_DeleteApply(t *testing.T) {
func TestSession_DeleteApply(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -133,7 +134,101 @@ func TestSessionEndpoint_DeleteApply(t *testing.T) {
}
}
func TestSessionEndpoint_Get(t *testing.T) {
func TestSession_Apply_ACLDeny(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Create the ACL.
req := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
session "foo" {
policy = "write"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var token string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &req, &token); err != nil {
t.Fatalf("err: %v", err)
}
// Just add a node.
s1.fsm.State().EnsureNode(1, &structs.Node{Node: "foo", Address: "127.0.0.1"})
// Try to create without a token, which will go through since version 8
// enforcement isn't enabled.
arg := structs.SessionRequest{
Datacenter: "dc1",
Op: structs.SessionCreate,
Session: structs.Session{
Node: "foo",
Name: "my-session",
},
}
var id1 string
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &id1); err != nil {
t.Fatalf("err: %v", err)
}
// Now turn on version 8 enforcement and try again, it should be denied.
var id2 string
s1.config.ACLEnforceVersion8 = true
err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &id2)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Now set a token and try again. This should go through.
arg.Token = token
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &id2); err != nil {
t.Fatalf("err: %v", err)
}
// Do a delete on the first session with version 8 enforcement off and
// no token. This should go through.
var out string
s1.config.ACLEnforceVersion8 = false
arg.Op = structs.SessionDestroy
arg.Token = ""
arg.Session.ID = id1
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &out); err != nil {
t.Fatalf("err: %v", err)
}
// Turn on version 8 enforcement and make sure the delete of the second
// session fails.
s1.config.ACLEnforceVersion8 = true
arg.Session.ID = id2
err = msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &out)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Now set a token and try again. This should go through.
arg.Token = token
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &out); err != nil {
t.Fatalf("err: %v", err)
}
}
func TestSession_Get(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -176,7 +271,7 @@ func TestSessionEndpoint_Get(t *testing.T) {
}
}
func TestSessionEndpoint_List(t *testing.T) {
func TestSession_List(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -227,7 +322,175 @@ func TestSessionEndpoint_List(t *testing.T) {
}
}
func TestSessionEndpoint_ApplyTimers(t *testing.T) {
func TestSession_Get_List_NodeSessions_ACLFilter(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Create the ACL.
req := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
session "foo" {
policy = "read"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var token string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &req, &token); err != nil {
t.Fatalf("err: %v", err)
}
// Create a node and a session.
s1.fsm.State().EnsureNode(1, &structs.Node{Node: "foo", Address: "127.0.0.1"})
arg := structs.SessionRequest{
Datacenter: "dc1",
Op: structs.SessionCreate,
Session: structs.Session{
Node: "foo",
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var out string
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &out); err != nil {
t.Fatalf("err: %v", err)
}
// Perform all the read operations, which should go through since version
// 8 ACL enforcement isn't enabled.
getR := structs.SessionSpecificRequest{
Datacenter: "dc1",
Session: out,
}
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.Get", &getR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 1 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
listR := structs.DCSpecificRequest{
Datacenter: "dc1",
}
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.List", &listR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 1 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
nodeR := structs.NodeSpecificRequest{
Datacenter: "dc1",
Node: "foo",
}
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.NodeSessions", &nodeR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 1 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
// Now turn on version 8 enforcement and make sure everything is empty.
s1.config.ACLEnforceVersion8 = true
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.Get", &getR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 0 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.List", &listR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 0 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.NodeSessions", &nodeR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 0 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
// Finally, supply the token and make sure the reads are allowed.
getR.Token = token
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.Get", &getR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 1 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
listR.Token = token
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.List", &listR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 1 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
nodeR.Token = token
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.NodeSessions", &nodeR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 1 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
// Try to get a session that doesn't exist to make sure that's handled
// correctly by the filter (it will get passed a nil slice).
getR.Session = "adf4238a-882b-9ddc-4a9d-5b6758e4159e"
{
var sessions structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.Get", &getR, &sessions); err != nil {
t.Fatalf("err: %v", err)
}
if len(sessions.Sessions) != 0 {
t.Fatalf("bad: %v", sessions.Sessions)
}
}
}
func TestSession_ApplyTimers(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -268,7 +531,7 @@ func TestSessionEndpoint_ApplyTimers(t *testing.T) {
}
}
func TestSessionEndpoint_Renew(t *testing.T) {
func TestSession_Renew(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -428,7 +691,85 @@ func TestSessionEndpoint_Renew(t *testing.T) {
}
}
func TestSessionEndpoint_NodeSessions(t *testing.T) {
func TestSession_Renew_ACLDeny(t *testing.T) {
dir1, s1 := testServerWithConfig(t, func(c *Config) {
c.ACLDatacenter = "dc1"
c.ACLMasterToken = "root"
c.ACLDefaultPolicy = "deny"
c.ACLEnforceVersion8 = false
})
defer os.RemoveAll(dir1)
defer s1.Shutdown()
codec := rpcClient(t, s1)
defer codec.Close()
testutil.WaitForLeader(t, s1.RPC, "dc1")
// Create the ACL.
req := structs.ACLRequest{
Datacenter: "dc1",
Op: structs.ACLSet,
ACL: structs.ACL{
Name: "User token",
Type: structs.ACLTypeClient,
Rules: `
session "foo" {
policy = "write"
}
`,
},
WriteRequest: structs.WriteRequest{Token: "root"},
}
var token string
if err := msgpackrpc.CallWithCodec(codec, "ACL.Apply", &req, &token); err != nil {
t.Fatalf("err: %v", err)
}
// Just add a node.
s1.fsm.State().EnsureNode(1, &structs.Node{Node: "foo", Address: "127.0.0.1"})
// Create a session. The token won't matter here since we don't have
// version 8 ACL enforcement on yet.
arg := structs.SessionRequest{
Datacenter: "dc1",
Op: structs.SessionCreate,
Session: structs.Session{
Node: "foo",
Name: "my-session",
},
}
var id string
if err := msgpackrpc.CallWithCodec(codec, "Session.Apply", &arg, &id); err != nil {
t.Fatalf("err: %v", err)
}
// Renew without a token should go through without version 8 ACL
// enforcement.
renewR := structs.SessionSpecificRequest{
Datacenter: "dc1",
Session: id,
}
var session structs.IndexedSessions
if err := msgpackrpc.CallWithCodec(codec, "Session.Renew", &renewR, &session); err != nil {
t.Fatalf("err: %v", err)
}
// Now turn on version 8 enforcement and the renew should be rejected.
s1.config.ACLEnforceVersion8 = true
err := msgpackrpc.CallWithCodec(codec, "Session.Renew", &renewR, &session)
if err == nil || !strings.Contains(err.Error(), permissionDenied) {
t.Fatalf("err: %v", err)
}
// Set the token and it should go through.
renewR.Token = token
if err := msgpackrpc.CallWithCodec(codec, "Session.Renew", &renewR, &session); err != nil {
t.Fatalf("err: %v", err)
}
}
func TestSession_NodeSessions(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()
@ -486,7 +827,7 @@ func TestSessionEndpoint_NodeSessions(t *testing.T) {
}
}
func TestSessionEndpoint_Apply_BadTTL(t *testing.T) {
func TestSession_Apply_BadTTL(t *testing.T) {
dir1, s1 := testServer(t)
defer os.RemoveAll(dir1)
defer s1.Shutdown()

9
consul/state/schema.go
View File

@ -78,6 +78,15 @@ func nodesTableSchema() *memdb.TableSchema {
Lowercase: true,
},
},
"meta": &memdb.IndexSchema{
Name: "meta",
AllowMissing: true,
Unique: false,
Indexer: &memdb.StringMapFieldIndex{
Field: "Meta",
Lowercase: false,
},
},
},
}
}

142
consul/state/state_store.go
View File

@ -349,9 +349,9 @@ func (s *StateStore) getWatchTables(method string) []string {
return []string{"nodes"}
case "Services":
return []string{"services"}
case "ServiceNodes", "NodeServices":
case "NodeService", "NodeServices", "ServiceNodes":
return []string{"nodes", "services"}
case "NodeChecks", "ServiceChecks", "ChecksInState":
case "NodeCheck", "NodeChecks", "ServiceChecks", "ChecksInState":
return []string{"checks"}
case "CheckServiceNodes", "NodeInfo", "NodeDump":
return []string{"nodes", "services", "checks"}
@ -426,6 +426,7 @@ func (s *StateStore) ensureRegistrationTxn(tx *memdb.Txn, idx uint64, watches *D
Node: req.Node,
Address: req.Address,
TaggedAddresses: req.TaggedAddresses,
Meta: req.NodeMeta,
}
if err := s.ensureNodeTxn(tx, idx, watches, node); err != nil {
return fmt.Errorf("failed inserting node: %s", err)
@ -438,6 +439,12 @@ func (s *StateStore) ensureRegistrationTxn(tx *memdb.Txn, idx uint64, watches *D
}
}
// TODO (slackpad) In Consul 0.8 ban checks that don't have the same
// node as the top-level registration. This is just weird to be able to
// update unrelated nodes' checks from in here. In 0.7.2 we banned this
// up in the ACL check since that's guarded behind an opt-in flag until
// Consul 0.8.
// Add the checks, if any.
if req.Check != nil {
if err := s.ensureCheckTxn(tx, idx, watches, req.Check); err != nil {
@ -542,6 +549,35 @@ func (s *StateStore) Nodes() (uint64, structs.Nodes, error) {
return idx, results, nil
}
// NodesByMeta is used to return all nodes with the given meta key/value pair.
func (s *StateStore) NodesByMeta(filters map[string]string) (uint64, structs.Nodes, error) {
if len(filters) > 1 {
return 0, nil, fmt.Errorf("multiple meta filters not supported")
}
tx := s.db.Txn(false)
defer tx.Abort()
// Get the table index.
idx := maxIndexTxn(tx, s.getWatchTables("Nodes")...)
// Retrieve all of the nodes
var args []interface{}
for key, value := range filters {
args = append(args, key, value)
}
nodes, err := tx.Get("nodes", "meta", args...)
if err != nil {
return 0, nil, fmt.Errorf("failed nodes lookup: %s", err)
}
// Create and return the nodes list.
var results structs.Nodes
for node := nodes.Next(); node != nil; node = nodes.Next() {
results = append(results, node.(*structs.Node))
}
return idx, results, nil
}
// DeleteNode is used to delete a given node by its ID.
func (s *StateStore) DeleteNode(idx uint64, nodeID string) error {
tx := s.db.Txn(true)
@ -752,6 +788,63 @@ func (s *StateStore) Services() (uint64, structs.Services, error) {
return idx, results, nil
}
// Services returns all services, filtered by the given node metadata.
func (s *StateStore) ServicesByNodeMeta(filters map[string]string) (uint64, structs.Services, error) {
if len(filters) > 1 {
return 0, nil, fmt.Errorf("multiple meta filters not supported")
}
tx := s.db.Txn(false)
defer tx.Abort()
// Get the table index.
idx := maxIndexTxn(tx, s.getWatchTables("ServiceNodes")...)
// Retrieve all of the nodes with the meta k/v pair
var args []interface{}
for key, value := range filters {
args = append(args, key, value)
}
nodes, err := tx.Get("nodes", "meta", args...)
if err != nil {
return 0, nil, fmt.Errorf("failed nodes lookup: %s", err)
}
// Populate the services map
unique := make(map[string]map[string]struct{})
for node := nodes.Next(); node != nil; node = nodes.Next() {
n := node.(*structs.Node)
// List all the services on the node
services, err := tx.Get("services", "node", n.Node)
if err != nil {
return 0, nil, fmt.Errorf("failed querying services: %s", err)
}
// Rip through the services and enumerate them and their unique set of
// tags.
for service := services.Next(); service != nil; service = services.Next() {
svc := service.(*structs.ServiceNode)
tags, ok := unique[svc.ServiceName]
if !ok {
unique[svc.ServiceName] = make(map[string]struct{})
tags = unique[svc.ServiceName]
}
for _, tag := range svc.ServiceTags {
tags[tag] = struct{}{}
}
}
}
// Generate the output structure.
var results = make(structs.Services)
for service, tags := range unique {
results[service] = make([]string, 0)
for tag, _ := range tags {
results[service] = append(results[service], tag)
}
}
return idx, results, nil
}
// ServiceNodes returns the nodes associated with a given service name.
func (s *StateStore) ServiceNodes(serviceName string) (uint64, structs.ServiceNodes, error) {
tx := s.db.Txn(false)
@ -848,12 +941,35 @@ func (s *StateStore) parseServiceNodes(tx *memdb.Txn, services structs.ServiceNo
node := n.(*structs.Node)
s.Address = node.Address
s.TaggedAddresses = node.TaggedAddresses
s.NodeMeta = node.Meta
results = append(results, s)
}
return results, nil
}
// NodeService is used to retrieve a specific service associated with the given
// node.
func (s *StateStore) NodeService(nodeID string, serviceID string) (uint64, *structs.NodeService, error) {
tx := s.db.Txn(false)
defer tx.Abort()
// Get the table index.
idx := maxIndexTxn(tx, s.getWatchTables("NodeService")...)
// Query the service
service, err := tx.First("services", "id", nodeID, serviceID)
if err != nil {
return 0, nil, fmt.Errorf("failed querying service for node %q: %s", nodeID, err)
}
if service != nil {
return idx, service.(*structs.ServiceNode).ToNodeService(), nil
} else {
return idx, nil, nil
}
}
// NodeServices is used to query service registrations by node ID.
func (s *StateStore) NodeServices(nodeID string) (uint64, *structs.NodeServices, error) {
tx := s.db.Txn(false)
@ -1056,6 +1172,27 @@ func (s *StateStore) ensureCheckTxn(tx *memdb.Txn, idx uint64, watches *DumbWatc
return nil
}
// NodeCheck is used to retrieve a specific check associated with the given
// node.
func (s *StateStore) NodeCheck(nodeID string, checkID types.CheckID) (uint64, *structs.HealthCheck, error) {
tx := s.db.Txn(false)
defer tx.Abort()
// Get the table index.
idx := maxIndexTxn(tx, s.getWatchTables("NodeCheck")...)
// Return the check.
check, err := tx.First("checks", "id", nodeID, string(checkID))
if err != nil {
return 0, nil, fmt.Errorf("failed check lookup: %s", err)
}
if check != nil {
return idx, check.(*structs.HealthCheck), nil
} else {
return idx, nil, nil
}
}
// NodeChecks is used to retrieve checks associated with the
// given node from the state store.
func (s *StateStore) NodeChecks(nodeID string) (uint64, structs.HealthChecks, error) {
@ -1343,6 +1480,7 @@ func (s *StateStore) parseNodes(tx *memdb.Txn, idx uint64,
Node: node.Node,
Address: node.Address,
TaggedAddresses: node.TaggedAddresses,
Meta: node.Meta,
}
// Query the node services

202
consul/state/state_store_test.go
View File

@ -241,6 +241,9 @@ func TestStateStore_EnsureRegistration(t *testing.T) {
TaggedAddresses: map[string]string{
"hello": "world",
},
NodeMeta: map[string]string{
"somekey": "somevalue",
},
}
if err := s.EnsureRegistration(1, req); err != nil {
t.Fatalf("err: %s", err)
@ -255,6 +258,7 @@ func TestStateStore_EnsureRegistration(t *testing.T) {
if out.Node != "node1" || out.Address != "1.2.3.4" ||
len(out.TaggedAddresses) != 1 ||
out.TaggedAddresses["hello"] != "world" ||
out.Meta["somekey"] != "somevalue" ||
out.CreateIndex != created || out.ModifyIndex != modified {
t.Fatalf("bad node returned: %#v", out)
}
@ -284,11 +288,24 @@ func TestStateStore_EnsureRegistration(t *testing.T) {
if len(out.Services) != 1 {
t.Fatalf("bad: %#v", out.Services)
}
s := out.Services["redis1"]
if s.ID != "redis1" || s.Service != "redis" ||
s.Address != "1.1.1.1" || s.Port != 8080 ||
s.CreateIndex != created || s.ModifyIndex != modified {
t.Fatalf("bad service returned: %#v", s)
r := out.Services["redis1"]
if r == nil || r.ID != "redis1" || r.Service != "redis" ||
r.Address != "1.1.1.1" || r.Port != 8080 ||
r.CreateIndex != created || r.ModifyIndex != modified {
t.Fatalf("bad service returned: %#v", r)
}
idx, r, err = s.NodeService("node1", "redis1")
if err != nil {
t.Fatalf("err: %s", err)
}
if idx != modified {
t.Fatalf("bad index: %d", idx)
}
if r == nil || r.ID != "redis1" || r.Service != "redis" ||
r.Address != "1.1.1.1" || r.Port != 8080 ||
r.CreateIndex != created || r.ModifyIndex != modified {
t.Fatalf("bad service returned: %#v", r)
}
}
verifyNode(1, 2)
@ -321,6 +338,18 @@ func TestStateStore_EnsureRegistration(t *testing.T) {
c.CreateIndex != created || c.ModifyIndex != modified {
t.Fatalf("bad check returned: %#v", c)
}
idx, c, err = s.NodeCheck("node1", "check1")
if err != nil {
t.Fatalf("err: %s", err)
}
if idx != modified {
t.Fatalf("bad index: %d", idx)
}
if c.Node != "node1" || c.CheckID != "check1" || c.Name != "check" ||
c.CreateIndex != created || c.ModifyIndex != modified {
t.Fatalf("bad check returned: %#v", c)
}
}
verifyNode(1, 3)
verifyService(2, 3)
@ -726,6 +755,97 @@ func BenchmarkGetNodes(b *testing.B) {
}
}
func TestStateStore_GetNodesByMeta(t *testing.T) {
s := testStateStore(t)
// Listing with no results returns nil
idx, res, err := s.NodesByMeta(map[string]string{"somekey": "somevalue"})
if idx != 0 || res != nil || err != nil {
t.Fatalf("expected (0, nil, nil), got: (%d, %#v, %#v)", idx, res, err)
}
// Create some nodes in the state store
node0 := &structs.Node{Node: "node0", Address: "127.0.0.1", Meta: map[string]string{"role": "client", "common": "1"}}
if err := s.EnsureNode(0, node0); err != nil {
t.Fatalf("err: %v", err)
}
node1 := &structs.Node{Node: "node1", Address: "127.0.0.1", Meta: map[string]string{"role": "server", "common": "1"}}
if err := s.EnsureNode(1, node1); err != nil {
t.Fatalf("err: %v", err)
}
// Retrieve the node with role=client
idx, nodes, err := s.NodesByMeta(map[string]string{"role": "client"})
if err != nil {
t.Fatalf("err: %s", err)
}
if idx != 1 {
t.Fatalf("bad index: %d", idx)
}
// Only one node was returned
if n := len(nodes); n != 1 {
t.Fatalf("bad node count: %d", n)
}
// Make sure the node is correct
if nodes[0].CreateIndex != 0 || nodes[0].ModifyIndex != 0 {
t.Fatalf("bad node index: %d, %d", nodes[0].CreateIndex, nodes[0].ModifyIndex)
}
if nodes[0].Node != "node0" {
t.Fatalf("bad: %#v", nodes[0])
}
if !reflect.DeepEqual(nodes[0].Meta, node0.Meta) {
t.Fatalf("bad: %v != %v", nodes[0].Meta, node0.Meta)
}
// Retrieve both nodes via their common meta field
idx, nodes, err = s.NodesByMeta(map[string]string{"common": "1"})
if err != nil {
t.Fatalf("err: %s", err)
}
if idx != 1 {
t.Fatalf("bad index: %d", idx)
}
// All nodes were returned
if n := len(nodes); n != 2 {
t.Fatalf("bad node count: %d", n)
}
// Make sure the nodes match
for i, node := range nodes {
if node.CreateIndex != uint64(i) || node.ModifyIndex != uint64(i) {
t.Fatalf("bad node index: %d, %d", node.CreateIndex, node.ModifyIndex)
}
name := fmt.Sprintf("node%d", i)
if node.Node != name {
t.Fatalf("bad: %#v", node)
}
if v, ok := node.Meta["common"]; !ok || v != "1" {
t.Fatalf("bad: %v", node.Meta)
}
}
}
func BenchmarkGetNodesByMeta(b *testing.B) {
s, err := NewStateStore(nil)
if err != nil {
b.Fatalf("err: %s", err)
}
if err := s.EnsureNode(100, &structs.Node{Node: "foo", Address: "127.0.0.1"}); err != nil {
b.Fatalf("err: %v", err)
}
if err := s.EnsureNode(101, &structs.Node{Node: "bar", Address: "127.0.0.2"}); err != nil {
b.Fatalf("err: %v", err)
}
for i := 0; i < b.N; i++ {
s.Nodes()
}
}
func TestStateStore_DeleteNode(t *testing.T) {
s := testStateStore(t)
@ -1036,6 +1156,78 @@ func TestStateStore_Services(t *testing.T) {
}
}
func TestStateStore_ServicesByNodeMeta(t *testing.T) {
s := testStateStore(t)
// Listing with no results returns nil
idx, res, err := s.ServicesByNodeMeta(map[string]string{"somekey": "somevalue"})
if idx != 0 || len(res) != 0 || err != nil {
t.Fatalf("expected (0, nil, nil), got: (%d, %#v, %#v)", idx, res, err)
}
// Create some nodes and services in the state store
node0 := &structs.Node{Node: "node0", Address: "127.0.0.1", Meta: map[string]string{"role": "client", "common": "1"}}
if err := s.EnsureNode(0, node0); err != nil {
t.Fatalf("err: %v", err)
}
node1 := &structs.Node{Node: "node1", Address: "127.0.0.1", Meta: map[string]string{"role": "server", "common": "1"}}
if err := s.EnsureNode(1, node1); err != nil {
t.Fatalf("err: %v", err)
}
ns1 := &structs.NodeService{
ID: "service1",
Service: "redis",
Tags: []string{"prod", "master"},
Address: "1.1.1.1",
Port: 1111,
}
if err := s.EnsureService(2, "node0", ns1); err != nil {
t.Fatalf("err: %s", err)
}
ns2 := &structs.NodeService{
ID: "service1",
Service: "redis",
Tags: []string{"prod", "slave"},
Address: "1.1.1.1",
Port: 1111,
}
if err := s.EnsureService(3, "node1", ns2); err != nil {
t.Fatalf("err: %s", err)
}
// Filter the services by the first node's meta value
idx, res, err = s.ServicesByNodeMeta(map[string]string{"role": "client"})
if err != nil {
t.Fatalf("err: %s", err)
}
if idx != 3 {
t.Fatalf("bad index: %d", idx)
}
expected := structs.Services{
"redis": []string{"prod", "master"},
}
if !reflect.DeepEqual(res, expected) {
t.Fatalf("bad: %v %v", res, expected)
}
// Get all services using the common meta value
idx, res, err = s.ServicesByNodeMeta(map[string]string{"common": "1"})
if err != nil {
t.Fatalf("err: %s", err)
}
if idx != 3 {
t.Fatalf("bad index: %d", idx)
}
expected = structs.Services{
"redis": []string{"prod", "master", "slave"},
}
sort.Strings(res["redis"])
sort.Strings(expected["redis"])
if !reflect.DeepEqual(res, expected) {
t.Fatalf("bad: %v %v", res, expected)
}
}
func TestStateStore_ServiceNodes(t *testing.T) {
s := testStateStore(t)

33
consul/structs/structs.go
View File

@ -173,6 +173,7 @@ type RegisterRequest struct {
Node string
Address string
TaggedAddresses map[string]string
NodeMeta map[string]string
Service *NodeService
Check *HealthCheck
Checks HealthChecks
@ -183,6 +184,26 @@ func (r *RegisterRequest) RequestDatacenter() string {
return r.Datacenter
}
// ChangesNode returns true if the given register request changes the given
// node, which can be nil. This only looks for changes to the node record itself,
// not any of the health checks.
func (r *RegisterRequest) ChangesNode(node *Node) bool {
// This means it's creating the node.
if node == nil {
return true
}
// Check if any of the node-level fields are being changed.
if r.Node != node.Node ||
r.Address != node.Address ||
!reflect.DeepEqual(r.TaggedAddresses, node.TaggedAddresses) ||
!reflect.DeepEqual(r.NodeMeta, node.Meta) {
return true
}
return false
}
// DeregisterRequest is used for the Catalog.Deregister endpoint
// to deregister a node as providing a service. If no service is
// provided the entire node is deregistered.
@ -208,8 +229,9 @@ type QuerySource struct {
// DCSpecificRequest is used to query about a specific DC
type DCSpecificRequest struct {
Datacenter string
Source QuerySource
Datacenter string
NodeMetaFilters map[string]string
Source QuerySource
QueryOptions
}
@ -259,6 +281,7 @@ type Node struct {
Node string
Address string
TaggedAddresses map[string]string
Meta map[string]string
RaftIndex
}
@ -268,8 +291,8 @@ type Nodes []*Node
// Maps service name to available tags
type Services map[string][]string
// ServiceNode represents a node that is part of a service. Address and
// TaggedAddresses are node-related fields that are always empty in the state
// ServiceNode represents a node that is part of a service. Address, TaggedAddresses,
// and NodeMeta are node-related fields that are always empty in the state
// store and are filled in on the way out by parseServiceNodes(). This is also
// why PartialClone() skips them, because we know they are blank already so it
// would be a waste of time to copy them.
@ -277,6 +300,7 @@ type ServiceNode struct {
Node string
Address string
TaggedAddresses map[string]string
NodeMeta map[string]string
ServiceID string
ServiceName string
ServiceTags []string
@ -469,6 +493,7 @@ type NodeInfo struct {
Node string
Address string
TaggedAddresses map[string]string
Meta map[string]string
Services []*NodeService
Checks HealthChecks
}

52
consul/structs/structs_test.go
View File

@ -105,6 +105,51 @@ func TestStructs_ACL_IsSame(t *testing.T) {
check(func() { other.Rules = "" }, func() { other.Rules = "service \"\" { policy = \"read\" }" })
}
func TestStructs_RegisterRequest_ChangesNode(t *testing.T) {
req := &RegisterRequest{
Node: "test",
Address: "127.0.0.1",
TaggedAddresses: make(map[string]string),
NodeMeta: map[string]string{
"role": "server",
},
}
node := &Node{
Node: "test",
Address: "127.0.0.1",
TaggedAddresses: make(map[string]string),
Meta: map[string]string{
"role": "server",
},
}
check := func(twiddle, restore func()) {
if req.ChangesNode(node) {
t.Fatalf("should not change")
}
twiddle()
if !req.ChangesNode(node) {
t.Fatalf("should change")
}
restore()
if req.ChangesNode(node) {
t.Fatalf("should not change")
}
}
check(func() { req.Node = "nope" }, func() { req.Node = "test" })
check(func() { req.Address = "127.0.0.2" }, func() { req.Address = "127.0.0.1" })
check(func() { req.TaggedAddresses["wan"] = "nope" }, func() { delete(req.TaggedAddresses, "wan") })
check(func() { req.NodeMeta["invalid"] = "nope" }, func() { delete(req.NodeMeta, "invalid") })
if !req.ChangesNode(nil) {
t.Fatalf("should change")
}
}
// testServiceNode gives a fully filled out ServiceNode instance.
func testServiceNode() *ServiceNode {
return &ServiceNode{
@ -113,6 +158,9 @@ func testServiceNode() *ServiceNode {
TaggedAddresses: map[string]string{
"hello": "world",
},
NodeMeta: map[string]string{
"tag": "value",
},
ServiceID: "service1",
ServiceName: "dogs",
ServiceTags: []string{"prod", "v1"},
@ -134,12 +182,13 @@ func TestStructs_ServiceNode_PartialClone(t *testing.T) {
// Make sure the parts that weren't supposed to be cloned didn't get
// copied over, then zero-value them out so we can do a DeepEqual() on
// the rest of the contents.
if clone.Address != "" || len(clone.TaggedAddresses) != 0 {
if clone.Address != "" || len(clone.TaggedAddresses) != 0 || len(clone.NodeMeta) != 0 {
t.Fatalf("bad: %v", clone)
}
sn.Address = ""
sn.TaggedAddresses = nil
sn.NodeMeta = nil
if !reflect.DeepEqual(sn, clone) {
t.Fatalf("bad: %v", clone)
}
@ -159,6 +208,7 @@ func TestStructs_ServiceNode_Conversions(t *testing.T) {
// them out before we do the compare.
sn.Address = ""
sn.TaggedAddresses = nil
sn.NodeMeta = nil
if !reflect.DeepEqual(sn, sn2) {
t.Fatalf("bad: %v", sn2)
}

84
terraform/aws/variables.tf
View File

@ -1,38 +1,45 @@
variable "platform" {
default = "ubuntu"
description = "The OS Platform"
default = "ubuntu"
description = "The OS Platform"
}
variable "user" {
default = {
ubuntu = "ubuntu"
rhel6 = "ec2-user"
centos6 = "centos"
centos7 = "centos"
rhel7 = "ec2-user"
}
default = {
ubuntu = "ubuntu"
rhel6 = "ec2-user"
centos6 = "centos"
centos7 = "centos"
rhel7 = "ec2-user"
}
}
variable "ami" {
description = "AWS AMI Id, if you change, make sure it is compatible with instance type, not all AMIs allow all instance types "
default = {
us-east-1-ubuntu = "ami-fce3c696"
us-west-1-ubuntu = "ami-a9a8e4c9"
us-west-2-ubuntu = "ami-9abea4fb"
eu-west-1-ubuntu = "ami-47a23a30"
eu-central-1-ubuntu = "ami-accff2b1"
ap-northeast-1-ubuntu = "ami-90815290"
ap-southeast-1-ubuntu = "ami-0accf458"
ap-southeast-2-ubuntu = "ami-1dc8b127"
us-east-1-rhel6 = "ami-0d28fe66"
us-west-2-rhel6 = "ami-3d3c0a0d"
us-east-1-centos6 = "ami-57cd8732"
us-west-2-centos6 = "ami-1255b321"
us-east-1-rhel7 = "ami-2051294a"
us-west-2-rhel7 = "ami-775e4f16"
us-east-1-centos7 = "ami-6d1c2007"
us-west-1-centos7 = "ami-af4333cf"
}
description = "AWS AMI Id, if you change, make sure it is compatible with instance type, not all AMIs allow all instance types "
default = {
us-east-1-ubuntu = "ami-fce3c696"
us-east-2-ubuntu = "ami-b7075dd2"
us-west-1-ubuntu = "ami-a9a8e4c9"
us-west-2-ubuntu = "ami-9abea4fb"
eu-west-1-ubuntu = "ami-47a23a30"
eu-central-1-ubuntu = "ami-accff2b1"
ap-northeast-1-ubuntu = "ami-90815290"
ap-northeast-2-ubuntu = "ami-58af6136"
ap-southeast-1-ubuntu = "ami-0accf458"
ap-southeast-2-ubuntu = "ami-1dc8b127"
us-east-1-rhel6 = "ami-0d28fe66"
us-east-2-rhel6 = "ami-aff2a9ca"
us-west-2-rhel6 = "ami-3d3c0a0d"
us-east-1-centos6 = "ami-57cd8732"
us-east-2-centos6 = "ami-c299c2a7"
us-west-2-centos6 = "ami-1255b321"
us-east-1-rhel7 = "ami-2051294a"
us-east-2-rhel7 = "ami-0a33696f"
us-west-2-rhel7 = "ami-775e4f16"
us-east-1-centos7 = "ami-6d1c2007"
us-east-2-centos7 = "ami-6a2d760f"
us-west-1-centos7 = "ami-af4333cf"
}
}
variable "service_conf" {
@ -44,6 +51,7 @@ variable "service_conf" {
rhel7 = "rhel_consul.service"
}
}
variable "service_conf_dest" {
default = {
ubuntu = "upstart.conf"
@ -55,29 +63,29 @@ variable "service_conf_dest" {
}
variable "key_name" {
description = "SSH key name in your AWS account for AWS instances."
description = "SSH key name in your AWS account for AWS instances."
}
variable "key_path" {
description = "Path to the private key specified by key_name."
description = "Path to the private key specified by key_name."
}
variable "region" {
default = "us-east-1"
description = "The region of AWS, for AMI lookups."
default = "us-east-1"
description = "The region of AWS, for AMI lookups."
}
variable "servers" {
default = "3"
description = "The number of Consul servers to launch."
default = "3"
description = "The number of Consul servers to launch."
}
variable "instance_type" {
default = "t2.micro"
description = "AWS Instance type, if you change, make sure it is compatible with AMI, not all AMIs allow all instance types "
default = "t2.micro"
description = "AWS Instance type, if you change, make sure it is compatible with AMI, not all AMIs allow all instance types "
}
variable "tagName" {
default = "consul"
description = "Name tag for the servers"
default = "consul"
description = "Name tag for the servers"
}

1
terraform/shared/scripts/rhel_consul.service
View File

@ -8,7 +8,6 @@ EnvironmentFile=-/etc/sysconfig/consul
Restart=on-failure
ExecStart=/usr/local/bin/consul agent $CONSUL_FLAGS -config-dir=/etc/systemd/system/consul.d
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGINT
[Install]
WantedBy=multi-user.target

31
testutil/server.go
View File

@ -23,15 +23,11 @@ import (
"os"
"os/exec"
"strings"
"sync/atomic"
"github.com/hashicorp/consul/consul/structs"
"github.com/hashicorp/go-cleanhttp"
)
// offset is used to atomically increment the port numbers.
var offset uint64
// TestPerformanceConfig configures the performance parameters.
type TestPerformanceConfig struct {
RaftMultiplier uint `json:"raft_multiplier,omitempty"`
@ -57,6 +53,7 @@ type TestAddressConfig struct {
// TestServerConfig is the main server configuration struct.
type TestServerConfig struct {
NodeName string `json:"node_name"`
NodeMeta map[string]string `json:"node_meta"`
Performance *TestPerformanceConfig `json:"performance,omitempty"`
Bootstrap bool `json:"bootstrap,omitempty"`
Server bool `json:"server,omitempty"`
@ -82,10 +79,8 @@ type ServerConfigCallback func(c *TestServerConfig)
// defaultServerConfig returns a new TestServerConfig struct
// with all of the listen ports incremented by one.
func defaultServerConfig() *TestServerConfig {
idx := int(atomic.AddUint64(&offset, 1))
return &TestServerConfig{
NodeName: fmt.Sprintf("node%d", idx),
NodeName: fmt.Sprintf("node%d", randomPort()),
DisableCheckpoint: true,
Performance: &TestPerformanceConfig{
RaftMultiplier: 1,
@ -96,16 +91,26 @@ func defaultServerConfig() *TestServerConfig {
Bind: "127.0.0.1",
Addresses: &TestAddressConfig{},
Ports: &TestPortConfig{
DNS: 20000 + idx,
HTTP: 21000 + idx,
RPC: 22000 + idx,
SerfLan: 23000 + idx,
SerfWan: 24000 + idx,
Server: 25000 + idx,
DNS: randomPort(),
HTTP: randomPort(),
RPC: randomPort(),
SerfLan: randomPort(),
SerfWan: randomPort(),
Server: randomPort(),
},
}
}
// randomPort asks the kernel for a random port to use.
func randomPort() int {
l, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
panic(err)
}
defer l.Close()
return l.Addr().(*net.TCPAddr).Port
}
// TestService is used to serialize a service definition.
type TestService struct {
ID string `json:",omitempty"`

7
ui/index.html
View File

@ -19,6 +19,13 @@
</head>
<body>
<noscript>
<center>
<h2>JavaScript Required</h2>
<p>Please enable JavaScript in your web browser to use Consul UI.</p>
</center>
</noscript>
<div class="wrapper">
<div class="container">
<div class="col-md-12">

2
vendor/github.com/hashicorp/go-memdb/README.md generated vendored
View File

@ -19,7 +19,7 @@ The database provides the following:
* Rich Indexing - Tables can support any number of indexes, which can be simple like
a single field index, or more advanced compound field indexes. Certain types like
UUID can be efficiently compressed from strings into byte indexes for reduces
UUID can be efficiently compressed from strings into byte indexes for reduced
storage requirements.
For the underlying immutable radix trees, see [go-immutable-radix](https://github.com/hashicorp/go-immutable-radix).

175
vendor/github.com/hashicorp/go-memdb/index.go generated vendored
View File

@ -9,15 +9,27 @@ import (
// Indexer is an interface used for defining indexes
type Indexer interface {
// FromObject is used to extract an index value from an
// object or to indicate that the index value is missing.
FromObject(raw interface{}) (bool, []byte, error)
// ExactFromArgs is used to build an exact index lookup
// based on arguments
FromArgs(args ...interface{}) ([]byte, error)
}
// SingleIndexer is an interface used for defining indexes
// generating a single entry per object
type SingleIndexer interface {
// FromObject is used to extract an index value from an
// object or to indicate that the index value is missing.
FromObject(raw interface{}) (bool, []byte, error)
}
// MultiIndexer is an interface used for defining indexes
// generating multiple entries per object
type MultiIndexer interface {
// FromObject is used to extract index values from an
// object or to indicate that the index value is missing.
FromObject(raw interface{}) (bool, [][]byte, error)
}
// PrefixIndexer can optionally be implemented for any
// indexes that support prefix based iteration. This may
// not apply to all indexes.
@ -88,6 +100,155 @@ func (s *StringFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error) {
return val, nil
}
// StringSliceFieldIndex is used to extract a field from an object
// using reflection and builds an index on that field.
type StringSliceFieldIndex struct {
Field string
Lowercase bool
}
func (s *StringSliceFieldIndex) FromObject(obj interface{}) (bool, [][]byte, error) {
v := reflect.ValueOf(obj)
v = reflect.Indirect(v) // Dereference the pointer if any
fv := v.FieldByName(s.Field)
if !fv.IsValid() {
return false, nil,
fmt.Errorf("field '%s' for %#v is invalid", s.Field, obj)
}
if fv.Kind() != reflect.Slice || fv.Type().Elem().Kind() != reflect.String {
return false, nil, fmt.Errorf("field '%s' is not a string slice", s.Field)
}
length := fv.Len()
vals := make([][]byte, 0, length)
for i := 0; i < fv.Len(); i++ {
val := fv.Index(i).String()
if val == "" {
continue
}
if s.Lowercase {
val = strings.ToLower(val)
}
// Add the null character as a terminator
val += "\x00"
vals = append(vals, []byte(val))
}
if len(vals) == 0 {
return false, nil, nil
}
return true, vals, nil
}
func (s *StringSliceFieldIndex) FromArgs(args ...interface{}) ([]byte, error) {
if len(args) != 1 {
return nil, fmt.Errorf("must provide only a single argument")
}
arg, ok := args[0].(string)
if !ok {
return nil, fmt.Errorf("argument must be a string: %#v", args[0])
}
if s.Lowercase {
arg = strings.ToLower(arg)
}
// Add the null character as a terminator
arg += "\x00"
return []byte(arg), nil
}
func (s *StringSliceFieldIndex) PrefixFromArgs(args ...interface{}) ([]byte, error) {
val, err := s.FromArgs(args...)
if err != nil {
return nil, err
}
// Strip the null terminator, the rest is a prefix
n := len(val)
if n > 0 {
return val[:n-1], nil
}
return val, nil
}
// StringMapFieldIndex is used to extract a field of type map[string]string
// from an object using reflection and builds an index on that field.
type StringMapFieldIndex struct {
Field string
Lowercase bool
}
var MapType = reflect.MapOf(reflect.TypeOf(""), reflect.TypeOf("")).Kind()
func (s *StringMapFieldIndex) FromObject(obj interface{}) (bool, [][]byte, error) {
v := reflect.ValueOf(obj)
v = reflect.Indirect(v) // Dereference the pointer if any
fv := v.FieldByName(s.Field)
if !fv.IsValid() {
return false, nil, fmt.Errorf("field '%s' for %#v is invalid", s.Field, obj)
}
if fv.Kind() != MapType {
return false, nil, fmt.Errorf("field '%s' is not a map[string]string", s.Field)
}
length := fv.Len()
vals := make([][]byte, 0, length)
for _, key := range fv.MapKeys() {
k := key.String()
if k == "" {
continue
}
val := fv.MapIndex(key).String()
if s.Lowercase {
k = strings.ToLower(k)
val = strings.ToLower(val)
}
// Add the null character as a terminator
k += "\x00" + val + "\x00"
vals = append(vals, []byte(k))
}
if len(vals) == 0 {
return false, nil, nil
}
return true, vals, nil
}
func (s *StringMapFieldIndex) FromArgs(args ...interface{}) ([]byte, error) {
if len(args) > 2 || len(args) == 0 {
return nil, fmt.Errorf("must provide one or two arguments")
}
key, ok := args[0].(string)
if !ok {
return nil, fmt.Errorf("argument must be a string: %#v", args[0])
}
if s.Lowercase {
key = strings.ToLower(key)
}
// Add the null character as a terminator
key += "\x00"
if len(args) == 2 {
val, ok := args[1].(string)
if !ok {
return nil, fmt.Errorf("argument must be a string: %#v", args[1])
}
if s.Lowercase {
val = strings.ToLower(val)
}
// Add the null character as a terminator
key += val + "\x00"
}
return []byte(key), nil
}
// UUIDFieldIndex is used to extract a field from an object
// using reflection and builds an index on that field by treating
// it as a UUID. This is an optimization to using a StringFieldIndex
@ -270,7 +431,11 @@ type CompoundIndex struct {
func (c *CompoundIndex) FromObject(raw interface{}) (bool, []byte, error) {
var out []byte
for i, idx := range c.Indexes {
for i, idxRaw := range c.Indexes {
idx, ok := idxRaw.(SingleIndexer)
if !ok {
return false, nil, fmt.Errorf("sub-index %d error: %s", i, "sub-index must be a SingleIndexer")
}
ok, val, err := idx.FromObject(raw)
if err != nil {
return false, nil, fmt.Errorf("sub-index %d error: %v", i, err)

9
vendor/github.com/hashicorp/go-memdb/schema.go generated vendored
View File

@ -46,6 +46,9 @@ func (s *TableSchema) Validate() error {
if !s.Indexes["id"].Unique {
return fmt.Errorf("id index must be unique")
}
if _, ok := s.Indexes["id"].Indexer.(SingleIndexer); !ok {
return fmt.Errorf("id index must be a SingleIndexer")
}
for name, index := range s.Indexes {
if name != index.Name {
return fmt.Errorf("index name mis-match for '%s'", name)
@ -72,5 +75,11 @@ func (s *IndexSchema) Validate() error {
if s.Indexer == nil {
return fmt.Errorf("missing index function for '%s'", s.Name)
}
switch s.Indexer.(type) {
case SingleIndexer:
case MultiIndexer:
default:
return fmt.Errorf("indexer for '%s' must be a SingleIndexer or MultiIndexer", s.Name)
}
return nil
}

88
vendor/github.com/hashicorp/go-memdb/txn.go generated vendored
View File

@ -148,7 +148,8 @@ func (txn *Txn) Insert(table string, obj interface{}) error {
// Get the primary ID of the object
idSchema := tableSchema.Indexes[id]
ok, idVal, err := idSchema.Indexer.FromObject(obj)
idIndexer := idSchema.Indexer.(SingleIndexer)
ok, idVal, err := idIndexer.FromObject(obj)
if err != nil {
return fmt.Errorf("failed to build primary index: %v", err)
}
@ -167,7 +168,19 @@ func (txn *Txn) Insert(table string, obj interface{}) error {
indexTxn := txn.writableIndex(table, name)
// Determine the new index value
ok, val, err := indexSchema.Indexer.FromObject(obj)
var (
ok bool
vals [][]byte
err error
)
switch indexer := indexSchema.Indexer.(type) {
case SingleIndexer:
var val []byte
ok, val, err = indexer.FromObject(obj)
vals = [][]byte{val}
case MultiIndexer:
ok, vals, err = indexer.FromObject(obj)
}
if err != nil {
return fmt.Errorf("failed to build index '%s': %v", name, err)
}
@ -176,28 +189,44 @@ func (txn *Txn) Insert(table string, obj interface{}) error {
// This is done by appending the primary key which must
// be unique anyways.
if ok && !indexSchema.Unique {
val = append(val, idVal...)
for i := range vals {
vals[i] = append(vals[i], idVal...)
}
}
// Handle the update by deleting from the index first
if update {
okExist, valExist, err := indexSchema.Indexer.FromObject(existing)
var (
okExist bool
valsExist [][]byte
err error
)
switch indexer := indexSchema.Indexer.(type) {
case SingleIndexer:
var valExist []byte
okExist, valExist, err = indexer.FromObject(existing)
valsExist = [][]byte{valExist}
case MultiIndexer:
okExist, valsExist, err = indexer.FromObject(existing)
}
if err != nil {
return fmt.Errorf("failed to build index '%s': %v", name, err)
}
if okExist {
// Handle non-unique index by computing a unique index.
// This is done by appending the primary key which must
// be unique anyways.
if !indexSchema.Unique {
valExist = append(valExist, idVal...)
}
for i, valExist := range valsExist {
// Handle non-unique index by computing a unique index.
// This is done by appending the primary key which must
// be unique anyways.
if !indexSchema.Unique {
valExist = append(valExist, idVal...)
}
// If we are writing to the same index with the same value,
// we can avoid the delete as the insert will overwrite the
// value anyways.
if !bytes.Equal(valExist, val) {
indexTxn.Delete(valExist)
// If we are writing to the same index with the same value,
// we can avoid the delete as the insert will overwrite the
// value anyways.
if i >= len(vals) || !bytes.Equal(valExist, vals[i]) {
indexTxn.Delete(valExist)
}
}
}
}
@ -213,7 +242,9 @@ func (txn *Txn) Insert(table string, obj interface{}) error {
}
// Update the value of the index
indexTxn.Insert(val, obj)
for _, val := range vals {
indexTxn.Insert(val, obj)
}
}
return nil
}
@ -233,7 +264,8 @@ func (txn *Txn) Delete(table string, obj interface{}) error {
// Get the primary ID of the object
idSchema := tableSchema.Indexes[id]
ok, idVal, err := idSchema.Indexer.FromObject(obj)
idIndexer := idSchema.Indexer.(SingleIndexer)
ok, idVal, err := idIndexer.FromObject(obj)
if err != nil {
return fmt.Errorf("failed to build primary index: %v", err)
}
@ -253,7 +285,19 @@ func (txn *Txn) Delete(table string, obj interface{}) error {
indexTxn := txn.writableIndex(table, name)
// Handle the update by deleting from the index first
ok, val, err := indexSchema.Indexer.FromObject(existing)
var (
ok bool
vals [][]byte
err error
)
switch indexer := indexSchema.Indexer.(type) {
case SingleIndexer:
var val []byte
ok, val, err = indexer.FromObject(existing)
vals = [][]byte{val}
case MultiIndexer:
ok, vals, err = indexer.FromObject(existing)
}
if err != nil {
return fmt.Errorf("failed to build index '%s': %v", name, err)
}
@ -261,10 +305,12 @@ func (txn *Txn) Delete(table string, obj interface{}) error {
// Handle non-unique index by computing a unique index.
// This is done by appending the primary key which must
// be unique anyways.
if !indexSchema.Unique {
val = append(val, idVal...)
for _, val := range vals {
if !indexSchema.Unique {
val = append(val, idVal...)
}
indexTxn.Delete(val)
}
indexTxn.Delete(val)
}
}
return nil

22
vendor/github.com/hashicorp/memberlist/config.go generated vendored
View File

@ -126,8 +126,12 @@ type Config struct {
// per GossipInterval. Increasing this number causes the gossip messages
// to propagate across the cluster more quickly at the expense of
// increased bandwidth.
GossipInterval time.Duration
GossipNodes int
//
// GossipToTheDeadTime is the interval after which a node has died that
// we will still try to gossip to it. This gives it a chance to refute.
GossipInterval time.Duration
GossipNodes int
GossipToTheDeadTime time.Duration
// EnableCompression is used to control message compression. This can
// be used to reduce bandwidth usage at the cost of slightly more CPU
@ -179,6 +183,11 @@ type Config struct {
// behavior for using LogOutput. You cannot specify both LogOutput and Logger
// at the same time.
Logger *log.Logger
// Size of Memberlist's internal channel which handles UDP messages. The
// size of this determines the size of the queue which Memberlist will keep
// while UDP messages are handled.
HandoffQueueDepth int
}
// DefaultLANConfig returns a sane set of configurations for Memberlist.
@ -207,8 +216,9 @@ func DefaultLANConfig() *Config {
DisableTcpPings: false, // TCP pings are safe, even with mixed versions
AwarenessMaxMultiplier: 8, // Probe interval backs off to 8 seconds
GossipNodes: 3, // Gossip to 3 nodes
GossipInterval: 200 * time.Millisecond, // Gossip more rapidly
GossipNodes: 3, // Gossip to 3 nodes
GossipInterval: 200 * time.Millisecond, // Gossip more rapidly
GossipToTheDeadTime: 30 * time.Second, // Same as push/pull
EnableCompression: true, // Enable compression by default
@ -216,6 +226,8 @@ func DefaultLANConfig() *Config {
Keyring: nil,
DNSConfigPath: "/etc/resolv.conf",
HandoffQueueDepth: 1024,
}
}
@ -231,6 +243,7 @@ func DefaultWANConfig() *Config {
conf.ProbeInterval = 5 * time.Second
conf.GossipNodes = 4 // Gossip less frequently, but to an additional node
conf.GossipInterval = 500 * time.Millisecond
conf.GossipToTheDeadTime = 60 * time.Second
return conf
}
@ -247,6 +260,7 @@ func DefaultLocalConfig() *Config {
conf.ProbeTimeout = 200 * time.Millisecond
conf.ProbeInterval = time.Second
conf.GossipInterval = 100 * time.Millisecond
conf.GossipToTheDeadTime = 15 * time.Second
return conf
}

6
vendor/github.com/hashicorp/memberlist/memberlist.go generated vendored
View File

@ -129,7 +129,7 @@ func newMemberlist(conf *Config) (*Memberlist, error) {
leaveBroadcast: make(chan struct{}, 1),
udpListener: udpLn,
tcpListener: tcpLn,
handoff: make(chan msgHandoff, 1024),
handoff: make(chan msgHandoff, conf.HandoffQueueDepth),
nodeMap: make(map[string]*nodeState),
nodeTimers: make(map[string]*suspicion),
awareness: newAwareness(conf.AwarenessMaxMultiplier),
@ -496,7 +496,7 @@ func (m *Memberlist) SendTo(to net.Addr, msg []byte) error {
buf = append(buf, msg...)
// Send the message
return m.rawSendMsgUDP(to, buf)
return m.rawSendMsgUDP(to, nil, buf)
}
// SendToUDP is used to directly send a message to another node, without
@ -513,7 +513,7 @@ func (m *Memberlist) SendToUDP(to *Node, msg []byte) error {
// Send the message
destAddr := &net.UDPAddr{IP: to.Addr, Port: int(to.Port)}
return m.rawSendMsgUDP(destAddr, buf)
return m.rawSendMsgUDP(destAddr, to, buf)
}
// SendToTCP is used to directly send a message to another node, without

51
vendor/github.com/hashicorp/memberlist/net.go generated vendored
View File

@ -5,6 +5,7 @@ import (
"bytes"
"encoding/binary"
"fmt"
"hash/crc32"
"io"
"net"
"time"
@ -32,7 +33,7 @@ const (
// understand version 4 or greater.
ProtocolVersion2Compatible = 2
ProtocolVersionMax = 4
ProtocolVersionMax = 5
)
// messageType is an integer ID of a type of message that can be received
@ -53,6 +54,7 @@ const (
compressMsg
encryptMsg
nackRespMsg
hasCrcMsg
)
// compressionType is used to specify the compression algorithm
@ -338,8 +340,18 @@ func (m *Memberlist) ingestPacket(buf []byte, from net.Addr, timestamp time.Time
buf = plain
}
// Handle the command
m.handleCommand(buf, from, timestamp)
// See if there's a checksum included to verify the contents of the message
if len(buf) >= 5 && messageType(buf[0]) == hasCrcMsg {
crc := crc32.ChecksumIEEE(buf[5:])
expected := binary.BigEndian.Uint32(buf[1:5])
if crc != expected {
m.logger.Printf("[WARN] memberlist: Got invalid checksum for UDP packet: %x, %x", crc, expected)
return
}
m.handleCommand(buf[5:], from, timestamp)
} else {
m.handleCommand(buf, from, timestamp)
}
}
func (m *Memberlist) handleCommand(buf []byte, from net.Addr, timestamp time.Time) {
@ -601,7 +613,7 @@ func (m *Memberlist) sendMsg(to net.Addr, msg []byte) error {
// Fast path if nothing to piggypack
if len(extra) == 0 {
return m.rawSendMsgUDP(to, msg)
return m.rawSendMsgUDP(to, nil, msg)
}
// Join all the messages
@ -613,11 +625,11 @@ func (m *Memberlist) sendMsg(to net.Addr, msg []byte) error {
compound := makeCompoundMessage(msgs)
// Send the message
return m.rawSendMsgUDP(to, compound.Bytes())
return m.rawSendMsgUDP(to, nil, compound.Bytes())
}
// rawSendMsgUDP is used to send a UDP message to another host without modification
func (m *Memberlist) rawSendMsgUDP(to net.Addr, msg []byte) error {
func (m *Memberlist) rawSendMsgUDP(addr net.Addr, node *Node, msg []byte) error {
// Check if we have compression enabled
if m.config.EnableCompression {
buf, err := compressPayload(msg)
@ -631,6 +643,31 @@ func (m *Memberlist) rawSendMsgUDP(to net.Addr, msg []byte) error {
}
}
// Try to look up the destination node
if node == nil {
toAddr, _, err := net.SplitHostPort(addr.String())
if err != nil {
m.logger.Printf("[ERR] memberlist: Failed to parse address %q: %v", addr.String(), err)
return err
}
m.nodeLock.RLock()
nodeState, ok := m.nodeMap[toAddr]
m.nodeLock.RUnlock()
if ok {
node = &nodeState.Node
}
}
// Add a CRC to the end of the payload if the recipient understands
// ProtocolVersion >= 5
if node != nil && node.PMax >= 5 {
crc := crc32.ChecksumIEEE(msg)
header := make([]byte, 5, 5+len(msg))
header[0] = byte(hasCrcMsg)
binary.BigEndian.PutUint32(header[1:], crc)
msg = append(header, msg...)
}
// Check if we have encryption enabled
if m.config.EncryptionEnabled() {
// Encrypt the payload
@ -645,7 +682,7 @@ func (m *Memberlist) rawSendMsgUDP(to net.Addr, msg []byte) error {
}
metrics.IncrCounter([]string{"memberlist", "udp", "sent"}, float32(len(msg)))
_, err := m.udpListener.WriteTo(msg, to)
_, err := m.udpListener.WriteTo(msg, addr)
return err
}

37
vendor/github.com/hashicorp/memberlist/state.go generated vendored
View File

@ -261,7 +261,7 @@ func (m *Memberlist) probeNode(node *nodeState) {
}
compound := makeCompoundMessage(msgs)
if err := m.rawSendMsgUDP(destAddr, compound.Bytes()); err != nil {
if err := m.rawSendMsgUDP(destAddr, &node.Node, compound.Bytes()); err != nil {
m.logger.Printf("[ERR] memberlist: Failed to send compound ping and suspect message to %s: %s", destAddr, err)
return
}
@ -310,8 +310,11 @@ func (m *Memberlist) probeNode(node *nodeState) {
// Get some random live nodes.
m.nodeLock.RLock()
excludes := []string{m.config.Name, node.Name}
kNodes := kRandomNodes(m.config.IndirectChecks, excludes, m.nodes)
kNodes := kRandomNodes(m.config.IndirectChecks, m.nodes, func(n *nodeState) bool {
return n.Name == m.config.Name ||
n.Name == node.Name ||
n.State != stateAlive
})
m.nodeLock.RUnlock()
// Attempt an indirect ping.
@ -460,10 +463,24 @@ func (m *Memberlist) resetNodes() {
func (m *Memberlist) gossip() {
defer metrics.MeasureSince([]string{"memberlist", "gossip"}, time.Now())
// Get some random live nodes
// Get some random live, suspect, or recently dead nodes
m.nodeLock.RLock()
excludes := []string{m.config.Name}
kNodes := kRandomNodes(m.config.GossipNodes, excludes, m.nodes)
kNodes := kRandomNodes(m.config.GossipNodes, m.nodes, func(n *nodeState) bool {
if n.Name == m.config.Name {
return true
}
switch n.State {
case stateAlive, stateSuspect:
return false
case stateDead:
return time.Since(n.StateChange) > m.config.GossipToTheDeadTime
default:
return true
}
})
m.nodeLock.RUnlock()
// Compute the bytes available
@ -484,7 +501,7 @@ func (m *Memberlist) gossip() {
// Send the compound message
destAddr := &net.UDPAddr{IP: node.Addr, Port: int(node.Port)}
if err := m.rawSendMsgUDP(destAddr, compound.Bytes()); err != nil {
if err := m.rawSendMsgUDP(destAddr, &node.Node, compound.Bytes()); err != nil {
m.logger.Printf("[ERR] memberlist: Failed to send gossip to %s: %s", destAddr, err)
}
}
@ -497,8 +514,10 @@ func (m *Memberlist) gossip() {
func (m *Memberlist) pushPull() {
// Get a random live node
m.nodeLock.RLock()
excludes := []string{m.config.Name}
nodes := kRandomNodes(1, excludes, m.nodes)
nodes := kRandomNodes(1, m.nodes, func(n *nodeState) bool {
return n.Name == m.config.Name ||
n.State != stateAlive
})
m.nodeLock.RUnlock()
// If no nodes, bail

37
vendor/github.com/hashicorp/memberlist/util.go generated vendored
View File

@ -155,8 +155,9 @@ func randomOffset(n int) int {
// suspicionTimeout computes the timeout that should be used when
// a node is suspected
func suspicionTimeout(suspicionMult, n int, interval time.Duration) time.Duration {
nodeScale := math.Ceil(math.Log10(float64(n + 1)))
timeout := time.Duration(suspicionMult) * time.Duration(nodeScale) * interval
nodeScale := math.Max(1.0, math.Log10(math.Max(1.0, float64(n))))
// multiply by 1000 to keep some precision because time.Duration is an int64 type
timeout := time.Duration(suspicionMult) * time.Duration(nodeScale*1000) * interval / 1000
return timeout
}
@ -207,9 +208,10 @@ func moveDeadNodes(nodes []*nodeState) int {
return n - numDead
}
// kRandomNodes is used to select up to k random nodes, excluding a given
// node and any non-alive nodes. It is possible that less than k nodes are returned.
func kRandomNodes(k int, excludes []string, nodes []*nodeState) []*nodeState {
// kRandomNodes is used to select up to k random nodes, excluding any nodes where
// the filter function returns true. It is possible that less than k nodes are
// returned.
func kRandomNodes(k int, nodes []*nodeState, filterFn func(*nodeState) bool) []*nodeState {
n := len(nodes)
kNodes := make([]*nodeState, 0, k)
OUTER:
@ -221,16 +223,9 @@ OUTER:
idx := randomOffset(n)
node := nodes[idx]
// Exclude node if match
for _, exclude := range excludes {
if node.Name == exclude {
continue OUTER
}
}
// Exclude if not alive
if node.State != stateAlive {
continue
// Give the filter a shot at it.
if filterFn != nil && filterFn(node) {
continue OUTER
}
// Check if we have this node already
@ -327,10 +322,18 @@ func isLoopbackIP(ip_str string) bool {
return loopbackBlock.Contains(ip)
}
// Given a string of the form "host", "host:port", or "[ipv6::address]:port",
// Given a string of the form "host", "host:port",
// "ipv6::addr" or "[ipv6::address]:port",
// return true if the string includes a port.
func hasPort(s string) bool {
return strings.LastIndex(s, ":") > strings.LastIndex(s, "]")
last := strings.LastIndex(s, ":")
if last == -1 {
return false
}
if s[0] == '[' {
return s[last-1] == ']'
}
return strings.Index(s, ":") == last
}
// compressPayload takes an opaque input buffer, compresses it

12
vendor/vendor.json vendored
View File

@ -436,10 +436,10 @@
"revisionTime": "2016-06-09T02:05:29Z"
},
{
"checksumSHA1": "/V57CyN7x2NUlHoOzVL5GgGXX84=",
"checksumSHA1": "ZpTDFeRvXFwIvSHRD8eDYHxaj4Y=",
"path": "github.com/hashicorp/go-memdb",
"revision": "98f52f52d7a476958fa9da671354d270c50661a7",
"revisionTime": "2016-03-01T23:01:42Z"
"revision": "d2d2b77acab85aa635614ac17ea865969f56009e",
"revisionTime": "2017-01-07T16:22:14Z"
},
{
"checksumSHA1": "TNlVzNR1OaajcNi3CbQ3bGbaLGU=",
@ -568,10 +568,10 @@
"revisionTime": "2015-06-09T07:04:31Z"
},
{
"checksumSHA1": "AY1/cRsuWpoJMG0J821TqFo9nDE=",
"checksumSHA1": "hSoH77pX3FyU6kkYqOOYmf3r55Y=",
"path": "github.com/hashicorp/memberlist",
"revision": "0c5ba075f8520c65572f001331a1a43b756e01d7",
"revisionTime": "2016-08-12T18:27:57Z"
"revision": "9800c50ab79c002353852a9b1095e9591b161513",
"revisionTime": "2016-12-13T23:44:46Z"
},
{
"checksumSHA1": "qnlqWJYV81ENr61SZk9c65R1mDo=",

2
version/version_base.go
View File

@ -7,7 +7,7 @@ package version
// adding new versions and pick a name that will follow "version_base.go".
func init() {
// The main version number that is being run at the moment.
Version = "0.7.2"
Version = "0.7.3"
// A pre-release marker for the version. If this is "" (empty string)
// then it means that it is a final release. Otherwise, this is a pre-release

6
website/config.rb
View File

@ -6,7 +6,7 @@ set :base_url, "https://www.consul.io/"
activate :hashicorp do |h|
h.name = "consul"
h.version = "0.7.1"
h.version = "0.7.2"
h.github_slug = "hashicorp/consul"
end
@ -41,6 +41,8 @@ helpers do
#
# @return [String]
def description_for(page)
return escape_html(page.data.description || "")
description = page.data.description || ""
description = description.gsub(/\n+/, " ")
return escape_html(description)
end
end

9
website/source/assets/stylesheets/_docs.scss
View File

@ -200,8 +200,15 @@ body.layout-intro{
text-decoration: none;
}
pre{
pre {
margin: 0 0 18px;
// This will force the code to scroll horizontally on small screens
// instead of wrapping.
code {
overflow-wrap: normal;
white-space: pre;
}
}
a{

2
website/source/docs/agent/encryption.html.markdown
View File

@ -53,7 +53,7 @@ $ consul agent -data-dir=/tmp/consul -config-file=encrypt.json
All nodes within a Consul cluster must share the same encryption key in
order to send and receive cluster information.
# RPC Encryption with TLS
## RPC Encryption with TLS
Consul supports using TLS to verify the authenticity of servers and clients. To enable this,
Consul requires that all clients and servers have key pairs that are generated by a single

9
website/source/docs/agent/http/agent.html.markdown
View File

@ -128,6 +128,8 @@ This endpoint is used to return the configuration and member information of the
Consul 0.7.0 and later also includes a snapshot of various operating statistics under the `Stats` key. These statistics are intended to help human operators for debugging and may change over time, so this part of the interface should not be consumed programmatically.
Consul 0.7.3 and later also includes a block of user-defined node metadata values under the `Meta` key. These are arbitrary key/value pairs defined in the [node meta](/docs/agent/options.html#_node_meta) section of the agent configuration.
It returns a JSON body like this:
```javascript
@ -194,6 +196,10 @@ It returns a JSON body like this:
"DelegateMin": 2,
"DelegateMax": 4,
"DelegateCur": 4
},
"Meta": {
"instance_type": "i2.xlarge",
"os_version": "ubuntu_16.04",
}
}
```
@ -477,7 +483,8 @@ information.
If `Check` is provided, only one of `Script`, `HTTP`, `TCP` or `TTL`
should be specified. `Script` and `HTTP` also require `Interval`. The
created check will be named `service:<ServiceId>`.
created check will be named `service:<ServiceId>`. The `Status` field
can be provided to specify the initial state of the health check.
In Consul 0.7 and later, checks that are associated with a service may
also contain an optional `DeregisterCriticalServiceAfter` field, which

28
website/source/docs/agent/http/catalog.html.markdown
View File

@ -44,6 +44,9 @@ body must look something like:
"lan": "192.168.10.10",
"wan": "10.0.10.10"
},
"NodeMeta": {
"somekey": "somevalue"
},
"Service": {
"ID": "redis1",
"Service": "redis",
@ -73,6 +76,10 @@ the node with the catalog. `TaggedAddresses` can be used in conjunction with the
option and the `wan` address. The `lan` address was added in Consul 0.7 to help find
the LAN address if address translation is enabled.
The `Meta` block was added in Consul 0.7.3 to enable associating arbitrary metadata
key/value pairs with a node for filtering purposes. For more information on node metadata,
see the [node meta](/docs/agent/options.html#_node_meta) section of the configuration page.
If the `Service` key is provided, the service will also be registered. If
`ID` is not provided, it will be defaulted to the value of the `Service.Service` property.
Only one service with a given `ID` may be present per node. The service `Tags`, `Address`,
@ -191,6 +198,10 @@ the node list in ascending order based on the estimated round trip
time from that node. Passing `?near=_agent` will use the agent's
node for the sort.
In Consul 0.7.3 and later, the optional `?node-meta=` parameter can be
provided with a desired node metadata key/value pair of the form `key:value`.
This will filter the results to nodes with that pair present.
It returns a JSON body like this:
```javascript
@ -201,6 +212,9 @@ It returns a JSON body like this:
"TaggedAddresses": {
"lan": "10.1.10.11",
"wan": "10.1.10.11"
},
"Meta": {
"instance_type": "t2.medium"
}
},
{
@ -209,6 +223,9 @@ It returns a JSON body like this:
"TaggedAddresses": {
"lan": "10.1.10.11",
"wan": "10.1.10.12"
},
"Meta": {
"instance_type": "t2.large"
}
}
]
@ -222,6 +239,10 @@ This endpoint is hit with a `GET` and returns the services registered
in a given DC. By default, the datacenter of the agent is queried;
however, the `dc` can be provided using the `?dc=` query parameter.
In Consul 0.7.3 and later, the optional `?node-meta=` parameter can be
provided with a desired node metadata key/value pair of the form `key:value`.
This will filter the results to services with that pair present.
It returns a JSON body like this:
```javascript
@ -265,6 +286,9 @@ It returns a JSON body like this:
"lan": "192.168.10.10",
"wan": "10.0.10.10"
},
"Meta": {
"instance_type": "t2.medium"
}
"CreateIndex": 51,
"ModifyIndex": 51,
"Node": "foobar",
@ -286,6 +310,7 @@ The returned fields are as follows:
- `Address`: IP address of the Consul node on which the service is registered
- `TaggedAddresses`: List of explicit LAN and WAN IP addresses for the agent
- `Meta`: Added in Consul 0.7.3, a list of user-defined metadata key/value pairs for the node
- `CreateIndex`: Internal index value representing when the service was created
- `ModifyIndex`: Last index that modified the service
- `Node`: Node name of the Consul node on which the service is registered
@ -313,6 +338,9 @@ It returns a JSON body like this:
"TaggedAddresses": {
"lan": "10.1.10.12",
"wan": "10.1.10.12"
},
"Meta": {
"instance_type": "t2.medium"
}
},
"Services": {

3
website/source/docs/agent/http/health.html.markdown
View File

@ -131,6 +131,9 @@ It returns a JSON body like this:
"TaggedAddresses": {
"lan": "10.1.10.12",
"wan": "10.1.10.12"
},
"Meta": {
"instance_type": "t2.medium"
}
},
"Service": {

49
website/source/docs/agent/options.html.markdown
View File

@ -53,6 +53,9 @@ The options below are all specified on the command-line.
address when being accessed from a remote datacenter if the remote datacenter is configured
with <a href="#translate_wan_addrs">`translate_wan_addrs`</a>.
~> **Notice:** The hosted version of Consul Enterprise will be deprecated on
March 7th, 2017. For details, see https://atlas.hashicorp.com/help/consul/alternatives
* <a name="_atlas"></a><a href="#_atlas">`-atlas`</a> - This flag
enables [Atlas](https://atlas.hashicorp.com) integration.
It is used to provide the Atlas infrastructure name and the SCADA connection. The format of
@ -279,6 +282,15 @@ will exit with an error at startup.
* <a name="_node"></a><a href="#_node">`-node`</a> - The name of this node in the cluster.
This must be unique within the cluster. By default this is the hostname of the machine.
* <a name="_node_meta"></a><a href="#_node_meta">`-node-meta`</a> - Available in Consul 0.7.3 and later,
this specifies an arbitrary metadata key/value pair to associate with the node, of the form `key:value`.
This can be specified multiple times. Node metadata pairs have the following restrictions:
- A maximum of 64 key/value pairs can be registered per node.
- Metadata keys must be between 1 and 128 characters (inclusive) in length
- Metadata keys must contain only alphanumeric, `-`, and `_` characters.
- Metadata keys must not begin with the `consul-` prefix; that is reserved for internal use by Consul.
- Metadata values must be between 0 and 512 (inclusive) characters in length.
* <a name="_pid_file"></a><a href="#_pid_file">`-pid-file`</a> - This flag provides the file
path for the agent to store its PID. This is useful for sending signals (for example, `SIGINT`
to close the agent or `SIGHUP` to update check definite
@ -408,6 +420,28 @@ Consul will not enable TLS for the HTTP API unless the `https` port has been ass
all operations, and "extend-cache" allows any cached ACLs to be used, ignoring their TTL
values. If a non-cached ACL is used, "extend-cache" acts like "deny".
* <a name="acl_agent_master_token"></a><a href="#acl_agent_master_token">`acl_agent_master_token`</a> -
Used to access <a href="/docs/agent/http/agent.html">agent endpoints</a> that require agent read
or write privileges even if Consul servers aren't present to validate any tokens. This should only
be used by operators during outages, regular ACL tokens should normally be used by applications.
This was added in Consul 0.7.2 and is only used when <a href="#acl_enforce_version_8">`acl_enforce_version_8`</a>
is set to true.
* <a name="acl_agent_token"></a><a href="#acl_agent_token">`acl_agent_token`</a> - Used for clients
and servers to perform internal operations to the service catalog. If this isn't specified, then
the <a href="#acl_token">`acl_token`</a> will be used. This was added in Consul 0.7.2.
<br><br>
For clients, this token must at least have write access to the node name it will register as. For
servers, this must have write access to all nodes that are expected to join the cluster, as well
as write access to the "consul" service, which will be registered automatically on its behalf.
* <a name="acl_enforce_version_8"></a><a href="#acl_enforce_version_8">`acl_enforce_version_8`</a> -
Used for clients and servers to determine if enforcement should occur for new ACL policies being
previewed before Consul 0.8. Added in Consul 0.7.2, this will default to false in versions of
Consul prior to 0.8, and will default to true in Consul 0.8 and later. This helps ease the
transition to the new ACL features by allowing policies to be in place before enforcement begins.
Please see the [ACL internals guide](/docs/internals/acl.html) for more details.
* <a name="acl_master_token"></a><a href="#acl_master_token">`acl_master_token`</a> - Only used
for servers in the [`acl_datacenter`](#acl_datacenter). This token will be created with management-level
permissions if it does not exist. It allows operators to bootstrap the ACL system
@ -664,6 +698,19 @@ Consul will not enable TLS for the HTTP API unless the `https` port has been ass
* <a name="node_name"></a><a href="#node_name">`node_name`</a> Equivalent to the
[`-node` command-line flag](#_node).
* <a name="node_meta"></a><a href="#node_meta">`node_meta`</a> Available in Consul 0.7.3 and later,
This object allows associating arbitrary metadata key/value pairs with the local node, which can
then be used for filtering results from certain catalog endpoints. See the
[`-node-meta` command-line flag](#_node_meta) for more information.
```javascript
{
"node_meta": {
"instance_type": "t2.medium"
}
}
```
* <a name="performance"></a><a href="#performance">`performance`</a> Available in Consul 0.7 and
later, this is a nested object that allows tuning the performance of different subsystems in
Consul. See the [Server Performance](/docs/guides/performance.html) guide for more details. The
@ -869,7 +916,7 @@ Consul will not enable TLS for the HTTP API unless the `https` port has been ass
* <a name="telemetry-circonus_check_display_name"</a><a href="#telemetry-circonus_check_display_name">`circonus_check_display_name`</a>
Specifies a name to give a check when it is created. This name is displayed in the Circonus UI Checks list. Available in Consul 0.7.2 and later.
* <a name="telemetry-circonus_check_tags"</a><a href="telemetry-circonus_check_tags">`circonus_check_tags`</a>
* <a name="telemetry-circonus_check_tags"</a><a href="#telemetry-circonus_check_tags">`circonus_check_tags`</a>
Comma separated list of additional tags to add to a check when it is created. Available in Consul 0.7.2 and later.
* <a name="telemetry-circonus_broker_id"></a><a href="#telemetry-circonus_broker_id">`circonus_broker_id`</a>

73
website/source/docs/commands/index.html.markdown
View File

@ -65,3 +65,76 @@ Options:
CONSUL_RPC_ADDR env variable.
-wan Joins a server to another server in the WAN pool
```
## Environment Variables
In addition to CLI flags, Consul reads environment variables for behavior
defaults. CLI flags always take precedence over environment variables, but it
is often helpful to use environment variables to configure the Consul agent,
particularly with configuration management and init systems.
These environment variables and their purpose are described below:
## `CONSUL_HTTP_ADDR`
This is the HTTP API address to the *local* Consul agent
(not the remote server) specified as a URI:
```
CONSUL_HTTP_ADDR=127.0.0.1:8500
```
or as a Unix socket path:
```
CONSUL_HTTP_ADDR=unix://var/run/consul_http.sock
```
### `CONSUL_HTTP_TOKEN`
This is the API access token required when access control lists (ACLs)
are enabled, for example:
```
CONSUL_HTTP_TOKEN=aba7cbe5-879b-999a-07cc-2efd9ac0ffe
```
### `CONSUL_HTTP_AUTH`
This specifies HTTP Basic access credentials as a username:password pair:
```
CONSUL_HTTP_AUTH=operations:JPIMCmhDHzTukgO6
```
### `CONSUL_HTTP_SSL`
This is a boolean value (default is false) that enables the HTTPS URI
scheme and SSL connections to the HTTP API:
```
CONSUL_HTTP_SSL=true
```
### `CONSUL_HTTP_SSL_VERIFY`
This is a boolean value (default true) to specify SSL certificate verification; setting this value to `false` is not recommended for production use. Example
for development purposes:
```
CONSUL_HTTP_SSL_VERIFY=false
```
### `CONSUL_RPC_ADDR`
This is the RPC interface address for the local agent specified as a URI:
```
CONSUL_RPC_ADDR=127.0.0.1:8300
```
or as a Unix socket path:
```
CONSUL_RPC_ADDR=unix://var/run/consul_rpc.sock
```

4
website/source/docs/commands/kv.html.markdown
View File

@ -31,7 +31,9 @@ Usage: consul kv <subcommand> [options] [args]
Subcommands:
delete Removes data from the KV store
export Exports part of the KV tree in JSON format
get Retrieves or lists data from the KV store
import Imports part of the KV tree in JSON format
put Sets or updates data in the KV store
```
@ -39,7 +41,9 @@ For more information, examples, and usage about a subcommand, click on the name
of the subcommand in the sidebar or one of the links below:
- [delete](/docs/commands/kv/delete.html)
- [export](/docs/commands/kv/export.html)
- [get](/docs/commands/kv/get.html)
- [import](/docs/commands/kv/import.html)
- [put](/docs/commands/kv/put.html)
## Basic Examples

31
website/source/docs/commands/kv/export.html.markdown.erb Normal file
View File

@ -0,0 +1,31 @@
---
layout: "docs"
page_title: "Commands: KV Export"
sidebar_current: "docs-commands-kv-export"
---
# Consul KV Export
Command: `consul kv export`
The `kv export` command is used to retrieve key-value pairs for the given
prefix from Consul's key-value store, and write a JSON representation to
stdout. This can be used with the command "consul kv import" to move entire
trees between Consul clusters.
## Usage
Usage: `consul kv export [PREFIX]`
#### API Options
<%= partial "docs/commands/http_api_options" %>
## Examples
To export the tree at "vault/" in the key value store:
```
$ consul kv export vault/
# JSON output
```

2
website/source/docs/commands/kv/get.html.markdown.erb
View File

@ -24,6 +24,8 @@ Usage: `consul kv get [options] [KEY_OR_PREFIX]`
#### KV Get Options
* `-base64` - Base 64 encode the value. The default value is false.
* `-detailed` - Provide additional metadata about the key in addition to the
value such as the ModifyIndex and any flags that may have been set on the key.
The default value is false.

46
website/source/docs/commands/kv/import.html.markdown.erb Normal file
View File

@ -0,0 +1,46 @@
---
layout: "docs"
page_title: "Commands: KV Import"
sidebar_current: "docs-commands-kv-import"
---
# Consul KV Import
Command: `consul kv import`
The `kv import` command is used to import KV pairs from the JSON representation
generated by the `kv export` command.
## Usage
Usage: `consul kv import [DATA]`
#### API Options
<%= partial "docs/commands/http_api_options" %>
## Examples
To import from a file, prepend the filename with `@`:
```
$ consul kv import @values.json
# Output
```
To import from stdin, use `-` as the data parameter:
```
$ cat values.json | consul kv import -
# Output
```
You can also pass the JSON directly, however care must be taken with shell
escaping:
```
$ consul kv import "$(cat values.json)"
# Output
```

11
website/source/docs/commands/kv/put.html.markdown.erb
View File

@ -24,6 +24,8 @@ Usage: `consul kv put [options] KEY [DATA]`
operation will create the key and obtain the lock. The session must already
exist and be specified via the -session flag. The default value is false.
* `-base64` - Treat the data as base 64 encoded. The default value is false.
* `-cas` - Perform a Check-And-Set operation. Specifying this value also
requires the -modify-index flag to be set. The default value is false.
@ -34,7 +36,7 @@ Usage: `consul kv put [options] KEY [DATA]`
* `-modify-index=<int>` - Unsigned integer representing the ModifyIndex of the
key. This is used in combination with the -cas flag.
* `-release` - Forfeit the lock on the key at the givne path. This requires the
* `-release` - Forfeit the lock on the key at the given path. This requires the
-session flag to be set. The key must be held by the session in order to be
unlocked. The default value is false.
@ -60,6 +62,13 @@ $ consul kv put redis/config/connections
Success! Data written to: redis/config/connections
```
If the `-base64` flag is set, the data will be decoded before writing:
```
$ consul kv put -base64 foo/encoded aGVsbG8gd29ybGQK
Success! Data written to: foo/encoded
```
!> **Be careful when overwriting data!** The above operation would overwrite
the value at the key to the empty value.

5
website/source/docs/guides/atlas.html.markdown
View File

@ -8,6 +8,9 @@ description: |-
# Atlas Integration
~> **Notice:** The hosted version of Consul Enterprise will be deprecated on
March 7th, 2017. For details, see https://atlas.hashicorp.com/help/consul/alternatives
[Atlas](https://atlas.hashicorp.com?utm_source=oss&utm_medium=guide-atlas&utm_campaign=consul) is a service provided by HashiCorp to deploy applications and manage infrastructure.
Starting with Consul 0.5, it is possible to integrate Consul with Atlas. Atlas is able to display the state of the Consul cluster in its dashboard and set up alerts based on health checks. Additionally, nodes can use Atlas to auto-join a Consul cluster without hardcoding any configuration.
@ -18,7 +21,7 @@ Atlas is able to securely retrieve data from nodes as Consul maintains a long-ru
To enable Atlas integration, you must specify the name of the Atlas infrastructure and the Atlas authentication
token in your Consul configuration. The Atlas infrastructure name can be set either with the [`-atlas` CLI flag](/docs/agent/options.html#_atlas) or with the [`atlas_infrastructure` configuration option](/docs/agent/options.html#atlas_infrastructure). The Atlas token is set with the [`-atlas-token` CLI flag](/docs/agent/options.html#_atlas_token),
[`atlas_token` configuration option](/docs/agent/options.html#atlas_token), or `ATLAS_TOKEN` environment variable.
[`atlas_token` configuration option](/docs/agent/options.html#atlas_token), or `ATLAS_TOKEN` environment variable.
To get an Atlas username and token, [create an account here](https://atlas.hashicorp.com/account/new?utm_source=oss&utm_medium=guide-atlas&utm_campaign=consul) and replace the respective values in your Consul configuration with your credentials.

2
website/source/docs/guides/bootstrapping.html.markdown
View File

@ -26,7 +26,7 @@ or specify no value at all. Only servers that specify a value will attempt to bo
We recommend 3 or 5 total servers per datacenter. A single server deployment is _**highly**_ discouraged
as data loss is inevitable in a failure scenario. Please refer to the
[deployment table](/docs/internals/consensus.html#toc_4) for more detail.
[deployment table](/docs/internals/consensus.html#deployment-table) for more detail.
Suppose we are starting a 3 server cluster. We can start `Node A`, `Node B`, and `Node C` with each
providing the `-bootstrap-expect 3` flag. Once the nodes are started, you should see a message like:

94
website/source/docs/internals/acl.html.markdown
View File

@ -84,7 +84,9 @@ datacenter servers to resolve even uncached tokens. This is enabled by setting a
[`acl_replication_token`](/docs/agent/options.html#acl_replication_token) in the
configuration on the servers in the non-authoritative datacenters. With replication
enabled, the servers will maintain a replica of the authoritative datacenter's full
set of ACLs on the non-authoritative servers.
set of ACLs on the non-authoritative servers. The ACL replication token needs to be
a valid ACL token with management privileges, it can also be the same as the master
ACL token.
Replication occurs with a background process that looks for new ACLs approximately
every 30 seconds. Replicated changes are written at a rate that's throttled to
@ -529,3 +531,93 @@ These differences are outlined in the table below:
<td>The captured token, client's token, or anonymous token is used to filter the results, as described above.</td>
</tr>
</table>
<a name="version_8_acls"></a>
## ACL Changes Coming in Consul 0.8
Consul 0.8 will feature complete ACL coverage for all of Consul. To ease the
transition to the new policies, a beta version of complete ACL support is
available starting in Consul 0.7.2.
Here's a summary of the upcoming changes:
* Agents now check `node` and `service` ACL policies for catalog-related operations
in `/v1/agent` endpoints, such as service and check registration and health check
updates.
* Agents enforce a new `agent` ACL policy for utility operations in `/v1/agent`
endpoints, such as joins and leaves.
* A new `node` ACL policy is enforced throughout Consul, providing a mechanism to
restrict registration and discovery of nodes by name. This also applies to
service discovery, so provides an additional dimension for controlling access to
services.
* A new `session` ACL policy controls the ability to create session objects by node
name.
* Anonymous prepared queries (non-templates without a `Name`) now require a valid
session, which ties their creation to the new `session` ACL policy.
* The existing `event` ACL policy has been applied to the `/v1/event/list` endpoint.
#### New Configuration Options
To enable beta support for complete ACL coverage, set the
[`acl_enforce_version_8`](/docs/agent/options.html#acl_enforce_version_8) configuration
option to `true` on Consul clients and servers.
Two new configuration options are used once complete ACLs are enabled:
* [`acl_agent_master_token`](/docs/agent/options.html#acl_agent_master_token) is used as
a special access token that has `agent` ACL policy `write` privileges on each agent where
it is configured. This token should only be used by operators during outages when Consul
servers aren't available to resolve ACL tokens. Applications should use regular ACL
tokens during normal operation.
* [`acl_agent_token`](/docs/agent/options.html#acl_agent_token) is used internally by
Consul agents to perform operations to the service catalog when registering themselves
or sending network coordinates to the servers.
<br>
<br>
For clients, this token must at least have `node` ACL policy `write` access to the node
name it will register as. For servers, this must have `node` ACL policy `write` access to
all nodes that are expected to join the cluster, as well as `service` ACL policy `write`
access to the `consul` service, which will be registered automatically on its behalf.
Since clients now resolve ACLs locally, the [`acl_down_policy`](/docs/agent/options.html#acl_down_policy)
now applies to Consul clients as well as Consul servers. This will determine what the
client will do in the event that the servers are down.
Consul clients *do not* need to have the [`acl_master_token`](/docs/agent/options.html#acl_agent_master_token)
or the [`acl_datacenter`](/docs/agent/options.html#acl_datacenter) configured. They will
contact the Consul servers to determine if ACLs are enabled. If they detect that ACLs are
not enabled, they will check at most every 2 minutes to see if they have become enabled, and
will start enforcing ACLs automatically.
#### New ACL Policies
The new `agent` ACL policy looks like this:
```
agent "<node name prefix>" {
policy = "<read|write|deny>"
}
```
This affects utility-related agent endpoints, such as `/v1/agent/self` and `/v1/agent/join`.
The new `node` ACL policy looks like this:
```
node "<node name prefix>" {
policy = "<read|write|deny>"
}
````
This affects node registration, node discovery, service discovery, and endpoints like
`/v1/agent/members`.
The new `session` ACL policy looks like this:
```
session "<node name prefix>" {
policy = "<read|write|deny>"
}
```
This affects all the of `/v1/session` endpoints.

6
website/source/downloads_tools.html.erb
View File

@ -102,6 +102,9 @@ description: |-
</p>
<ul>
<li>
<a href="https://github.com/jippi/hashi-ui">hashi-ui</a> - A modern user interface for the Consul and Nomad
</li>
<li>
<a href="http://www.cfg4j.org">cfg4j</a> - Configuration library for Java distributed apps. Reads and auto-updates configuration from Consul KVs (and others)
</li>
@ -111,6 +114,9 @@ description: |-
<li>
<a href="https://github.com/kelseyhightower/confd">confd</a> - Manage local application configuration files using templates and data from etcd or Consul
</li>
<li>
<a href="https://github.com/ncbi/consul-announcer">consul-announcer</a> - Command line wrapper for registering services in Consul
</li>
<li>
<a href="https://github.com/myENA/consul-backinator">consul-backinator</a> - Command line Consul KV backup and restoration utility
</li>

393
website/source/index.html.erb
View File

@ -1,207 +1,208 @@
---
description: Service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
description: |-
Consul is a highly available and distributed service discovery and key-value
store designed with support for the modern data center to make distributed
systems and configuration easy.
---
<!-- Main jumbotron for a primary marketing message or call to action -->
<div id="jumbotron-mask">
<div id="jumbotron">
<div class="container">
<div class="col-lg-6 col-md-6">
<h2 class="rls-l">
Service discovery and configuration made easy.
Distributed, highly available, and
datacenter-aware.
</h2>
</div>
<div class="jumbo-logo-wrap col-lg-offset-1 col-lg-5 col-md-6 hidden-xs hidden-sm">
<div class="jumbo-logo"></div>
</div>
<!-- <p><a class="btn btn-primary btn-lg">Learn more &raquo;</a></p> -->
</div>
<div class="jumbotron-dots"></div>
</div>
<div id="jumbotron">
<div class="container">
<div class="col-lg-6 col-md-6">
<h2 class="rls-l">
Service discovery and configuration made easy.
Distributed, highly available, and
datacenter-aware.
</h2>
</div>
<div class="jumbo-logo-wrap col-lg-offset-1 col-lg-5 col-md-6 hidden-xs hidden-sm">
<div class="jumbo-logo"></div>
</div>
<!-- <p><a class="btn btn-primary btn-lg">Learn more &raquo;</a></p> -->
</div>
<div class="jumbotron-dots"></div>
</div>
</div>
<!-- Make sure that any changes made to jumbotron are reproduced here as this takes up the space in the layout -->
<div id="jumbotron-mask-dummy" aria-hidden="true">
<div id="jumbotron">
<div class="container">
<div class="col-lg-6 col-md-6">
<h2 class="rls-l">
Service discovery and configuration made easy.
Distributed, highly available, and
datacenter-aware.
</h2>
</div>
<div class="jumbo-logo-wrap col-lg-offset-1 col-lg-5 col-md-6 hidden-xs hidden-sm">
<div class="jumbo-logo"></div>
</div>
<!-- <p><a class="btn btn-primary btn-lg">Learn more &raquo;</a></p> -->
</div>
<div class="jumbotron-dots"></div>
</div>
<div id="jumbotron">
<div class="container">
<div class="col-lg-6 col-md-6">
<h2 class="rls-l">
Service discovery and configuration made easy.
Distributed, highly available, and
datacenter-aware.
</h2>
</div>
<div class="jumbo-logo-wrap col-lg-offset-1 col-lg-5 col-md-6 hidden-xs hidden-sm">
<div class="jumbo-logo"></div>
</div>
<!-- <p><a class="btn btn-primary btn-lg">Learn more &raquo;</a></p> -->
</div>
<div class="jumbotron-dots"></div>
</div>
</div>
<div id="features">
<div class="container">
<div class="row double-row">
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn discovery"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Service Discovery</h2>
<p>
Consul makes it simple for services to register themselves
and to discover other services via a DNS or HTTP interface.
Register external services such as SaaS providers as well.</p>
</div>
</div>
</div>
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn health"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Failure Detection</h2>
<p>Pairing service discovery with health checking prevents routing requests to unhealthy hosts and enables services to easily provide circuit breakers.</p>
</div>
</div>
</div>
</div>
<div class="row double-row">
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn multi"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Multi Datacenter</h2>
<p>Consul scales to multiple datacenters out of the box with no complicated configuration. Look up services in other datacenters, or keep the request local.</p>
</div>
</div>
</div>
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn config"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Key/Value Storage</h2>
<p>Flexible key/value store for dynamic configuration, feature flagging, coordination, leader election and more. Long poll for near-instant notification of configuration changes.</p>
</div>
</div>
</div>
</div>
</div> <!-- /container -->
</div> <!-- /features -->
<div id="demos">
<div class="container">
<div class="terminals row">
<div class="col-xs-12 col-lg-12 explantion">
<h2>DNS Query Interface</h2>
<p>
Look up services using Consul's built-in DNS server. Support
existing infrastructure without any code change.
</p>
</div>
<div class="terminal-item col-xs-12 col-lg-12">
<div class="terminal">
<header>
<h4>Terminal</h4>
<ul class='shell-dots'>
<li></li>
<li></li>
<li></li>
</ul>
</header>
<div class="terminal-window">
<div class="terminal">
<div class="display">
<p class="command"><span class="txt-r">admin@hashicorp</span>: dig web-frontend.service.consul. ANY</p>
<p>; &lt;&lt;&gt;&gt; DiG 9.8.3-P1 &lt;&lt;&gt;&gt; web-frontend.service.consul. ANY</p>
<p>;; global options: +cmd</p>
<p> </p>
<p>;; Got answer:</p>
<p>;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 29981</p>
<p>;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0</p>
<p> </p>
<p>;; QUESTION SECTION:</p>
<p>;web-frontend.service.consul. IN ANY</p>
<p> </p>
<p>;; ANSWER SECTION:</p>
<p>web-frontend.service.consul. 0 IN A <span class="txt-p">10.0.3.83</span></p>
<p>web-frontend.service.consul. 0 IN A <span class="txt-p">10.0.1.109</span></p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: <span class="cursor">&nbsp;</span></p>
</div>
</div>
</div>
</div>
</div> <!-- /.terminal-item -->
<div class="col-xs-12 col-lg-12 explantion">
<h2>Key Value Storage</h2>
<p>
Consul provides a hierarchical key/value store with a simple HTTP API.
Managing configuration has never been simpler.
</p>
</div>
<div class="terminal-item col-xs-12 col-lg-12">
<div class="terminal">
<header>
<h4>Terminal</h4>
<ul class='shell-dots'>
<li></li>
<li></li>
<li></li>
</ul>
</header>
<div class="terminal-window">
<div class="terminal">
<div class="display">
<p class="command"><span class="txt-r">admin@hashicorp</span>: curl -X PUT -d 'bar' http://localhost:8500/v1/kv/foo</p>
<p>true</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: curl http://localhost:8500/v1/kv/foo</p>
<p>[</p>
<p> {</p>
<p> "CreateIndex": 100,</p>
<p> "ModifyIndex": 200,</p>
<p> "Key": "foo",</p>
<p> "Flags": 0,</p>
<p> "Value": <span class="txt-p">"YmFy"</span></p>
<p> }</p>
<p>]</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: echo "YmFy" | base64 --decode</p>
<p>bar</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: <span class="cursor">&nbsp;</span></p>
</div>
</div>
</div>
</div>
</div> <!-- /.terminal-item -->
</div>
</div>
</div><!-- /#demos -->
<div id="cta">
<div class="container">
<div class="row">
<div class="intro">
<div class="left col-xs-12 col-sm-offset-2 col-sm-4">
<p>The intro and getting started guide contain
a simple and approachable walkthrough for running Consul locally.</p>
</div>
<div class="col-xs-12 col-sm-6 col-sm-offset-0 right">
<a class="outline-btn purple" href="/intro/index.html">Read the intro &#187;</a>
</div>
</div>
</div>
</div>
<div class="container">
<div class="row double-row">
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn discovery"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Service Discovery</h2>
<p>
Consul makes it simple for services to register themselves
and to discover other services via a DNS or HTTP interface.
Register external services such as SaaS providers as well.
</p>
</div>
</div>
</div>
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn health"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Failure Detection</h2>
<p>Pairing service discovery with health checking prevents routing requests to unhealthy hosts and enables services to easily provide circuit breakers.</p>
</div>
</div>
</div>
</div>
<div class="row double-row">
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn multi"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Multi Datacenter</h2>
<p>Consul scales to multiple datacenters out of the box with no complicated configuration. Look up services in other datacenters, or keep the request local.</p>
</div>
</div>
</div>
<div class="col-lg-6 col-md-6">
<div class="row">
<div class="col-lg-5 col-md-5">
<span class="icn config"></span>
</div>
<div class="col-lg-7 col-md-7">
<h2>Key/Value Storage</h2>
<p>Flexible key/value store for dynamic configuration, feature flagging, coordination, leader election and more. Long poll for near-instant notification of configuration changes.</p>
</div>
</div>
</div>
</div>
</div>
<!-- /container -->
</div>
<!-- /features -->
<div id="demos">
<div class="container">
<div class="terminals row">
<div class="col-xs-12 col-lg-12 explantion">
<h2>DNS Query Interface</h2>
<p>
Look up services using Consul's built-in DNS server. Support
existing infrastructure without any code change.
</p>
</div>
<div class="terminal-item col-xs-12 col-lg-12">
<div class="terminal">
<header>
<h4>Terminal</h4>
<ul class='shell-dots'>
<li></li>
<li></li>
<li></li>
</ul>
</header>
<div class="terminal-window">
<div class="terminal">
<div class="display">
<p class="command"><span class="txt-r">admin@hashicorp</span>: dig web-frontend.service.consul. ANY</p>
<p>; &lt;&lt;&gt;&gt; DiG 9.8.3-P1 &lt;&lt;&gt;&gt; web-frontend.service.consul. ANY</p>
<p>;; global options: +cmd</p>
<p> </p>
<p>;; Got answer:</p>
<p>;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 29981</p>
<p>;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0</p>
<p> </p>
<p>;; QUESTION SECTION:</p>
<p>;web-frontend.service.consul. IN ANY</p>
<p> </p>
<p>;; ANSWER SECTION:</p>
<p>web-frontend.service.consul. 0 IN A <span class="txt-p">10.0.3.83</span></p>
<p>web-frontend.service.consul. 0 IN A <span class="txt-p">10.0.1.109</span></p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: <span class="cursor">&nbsp;</span></p>
</div>
</div>
</div>
</div>
</div>
<!-- /.terminal-item -->
<div class="col-xs-12 col-lg-12 explantion">
<h2>Key Value Storage</h2>
<p>
Consul provides a hierarchical key/value store with a simple HTTP API.
Managing configuration has never been simpler.
</p>
</div>
<div class="terminal-item col-xs-12 col-lg-12">
<div class="terminal">
<header>
<h4>Terminal</h4>
<ul class='shell-dots'>
<li></li>
<li></li>
<li></li>
</ul>
</header>
<div class="terminal-window">
<div class="terminal">
<div class="display">
<p class="command"><span class="txt-r">admin@hashicorp</span>: consul kv put foo bar</p>
<p>Success! Data written to: foo</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: consul kv get foo</p>
<p>bar</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: consul kv get -detailed foo</p>
<p>CreateIndex 5</p>
<p>Flags 0</p>
<p>Key foo</p>
<p>LockIndex 0</p>
<p>ModifyIndex 5</p>
<p>Session -</p>
<p>Value bar</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: consul kv delete foo</p>
<p>Success! Deleted key: foo</p>
<p class="command"><span class="txt-r">admin@hashicorp</span>: <span class="cursor">&nbsp;</span></p>
</div>
</div>
</div>
</div>
</div>
<!-- /.terminal-item -->
</div>
</div>
</div>
<!-- /#demos -->
<div id="cta">
<div class="container">
<div class="row">
<div class="intro">
<div class="left col-xs-12 col-sm-offset-2 col-sm-4">
<p>The intro and getting started guide contain
a simple and approachable walkthrough for running Consul locally.
</p>
</div>
<div class="col-xs-12 col-sm-6 col-sm-offset-0 right">
<a class="outline-btn purple" href="/intro/index.html">Read the intro &#187;</a>
</div>
</div>
</div>
</div>
</div>

177
website/source/intro/getting-started/kv.html.markdown
View File

@ -17,113 +17,118 @@ This step assumes you have at least one Consul agent already running.
## Simple Usage
To demonstrate how simple it is to get started, we will manipulate a few keys
in the K/V store.
To demonstrate how simple it is to get started, we will manipulate a few keys in
the K/V store. There are two ways to interact with the Consul KV store: via the
HTTP API and via the Consul KV CLI. The examples below show using the Consul KV
CLI because it is the easiest to get started. For more advanced integrations,
you may want to use the [Consul KV HTTP API][kv-api]
Querying the local agent we started in the [Run the Agent step](agent.html),
we can first verify that there are no existing keys in the k/v store:
First let us explore the KV store. We can ask Consul for the value of the key at
the path named `redis/config/minconns`:
Also check https://www.hashicorp.com/blog/consul-kv-cli.html for using the 'consul kv' from
the commandline.
```text
$ curl -v http://localhost:8500/v1/kv/?recurse
* About to connect() to localhost port 8500 (#0)
* Trying 127.0.0.1... connected
> GET /v1/kv/?recurse HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:8500
> Accept: */*
>
< HTTP/1.1 404 Not Found
< X-Consul-Index: 1
< Date: Fri, 11 Apr 2014 02:10:28 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact
* Closing connection #0
```sh
$ consul kv get redis/config/minconns
Error! No key exists at: redis/config/minconns
```
Since there are no keys, we get a 404 response back. Now, we can `PUT` a
few example keys:
As you can see, we get no result, which makes sense because there is no data in
the KV store. Next we can insert or "put" values into the KV store.
```
$ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1
true
$ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key2?flags=42
true
$ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/sub/key3
true
$ curl http://localhost:8500/v1/kv/?recurse
[{"CreateIndex":97,"ModifyIndex":97,"Key":"web/key1","Flags":0,"Value":"dGVzdA=="},
{"CreateIndex":98,"ModifyIndex":98,"Key":"web/key2","Flags":42,"Value":"dGVzdA=="},
{"CreateIndex":99,"ModifyIndex":99,"Key":"web/sub/key3","Flags":0,"Value":"dGVzdA=="}]
```sh
$ consul kv put redis/config/minconns 1
Success! Data written to: redis/config/minconns
$ consul kv put redis/config/maxconns 25
Success! Data written to: redis/config/maxconns
$ consul kv put -flags=42 redis/config/users/admin abcd1234
Success! Data written to: redis/config/users/admin
```
Here we have created 3 keys, each with the value of "test". Note that the
`Value` field returned is base64 encoded to allow non-UTF8 characters. For the
key "web/key2", we set a `flag` value of 42. All keys support setting a 64-bit
integer flag value. This is not used internally by Consul, but it can be used by
clients to add meaningful metadata to any KV.
Now that we have keys in the store, we can query for the value of individual
keys:
After setting the values, we then issued a `GET` request to retrieve multiple
keys using the `?recurse` parameter.
You can also fetch a single key just as easily:
```text
$ curl http://localhost:8500/v1/kv/web/key1
[{"CreateIndex":97,"ModifyIndex":97,"Key":"web/key1","Flags":0,"Value":"dGVzdA=="}]
```sh
$ consul kv get redis/config/minconns
1
```
Deleting keys is simple as well, accomplished by using the `DELETE` verb. We can
delete a single key by specifying the full path, or we can recursively delete all
keys under a root using "?recurse":
Consul retains additional metadata about the field, which is retrieved using the
`-detailed` flag:
```text
$ curl -X DELETE http://localhost:8500/v1/kv/web/sub?recurse
$ curl http://localhost:8500/v1/kv/web?recurse
[{"CreateIndex":97,"ModifyIndex":97,"Key":"web/key1","Flags":0,"Value":"dGVzdA=="},
{"CreateIndex":98,"ModifyIndex":98,"Key":"web/key2","Flags":42,"Value":"dGVzdA=="}]
```sh
$ consul kv get -detailed redis/config/minconns
CreateIndex 207
Flags 0
Key redis/config/minconns
LockIndex 0
ModifyIndex 207
Session -
Value 1
```
A key can be modified by issuing a `PUT` request to the same URI and
providing a different message body. Additionally, Consul provides a
Check-And-Set operation, enabling atomic key updates. This is done by
providing the `?cas=` parameter with the last `ModifyIndex` value from
the GET request. For example, suppose we wanted to update "web/key1":
For the key "redis/config/users/admin", we set a `flag` value of 42. All keys
support setting a 64-bit integer flag value. This is not used internally by
Consul, but it can be used by clients to add meaningful metadata to any KV.
```text
$ curl -X PUT -d 'newval' http://localhost:8500/v1/kv/web/key1?cas=97
true
$ curl -X PUT -d 'newval' http://localhost:8500/v1/kv/web/key1?cas=97
false
It is possible to list all the keys in the store using the `recurse` options.
Results will be returned in lexicographical order:
```sh
$ consul kv get -recurse
redis/config/maxconns:25
redis/config/minconns:1
redis/config/users/admin:abcd1234
```
In this case, the first CAS update succeeds because the `ModifyIndex` is 97.
However the second operation fails because the `ModifyIndex` is no longer 97.
To delete a key from the Consul KV store, issue a "delete" call:
We can also make use of the `ModifyIndex` to wait for a key's value to change.
For example, suppose we wanted to wait for key2 to be modified:
```text
$ curl "http://localhost:8500/v1/kv/web/key2?index=101&wait=5s"
[{"CreateIndex":98,"ModifyIndex":101,"Key":"web/key2","Flags":42,"Value":"dGVzdA=="}]
```sh
$ consul kv delete redis/config/minconns
Success! Deleted key: redis/config/minconns
```
By providing "?index=", we are asking to wait until the key has a `ModifyIndex` greater
than 101. However the "?wait=5s" parameter restricts the query to at most 5 seconds,
returning the current, unchanged value. This can be used to efficiently wait for
key modifications. Additionally, this same technique can be used to wait for a list
of keys, waiting only until any of the keys has a newer modification time.
It is also possible to delete an entire prefix using the `recurse` option:
```sh
$ consul kv delete -recurse redis
Success! Deleted keys with prefix: redis
```
To update the value of an existing key, "put" a value at the same path:
```sh
$ consul kv put foo bar
$ consul kv get foo
bar
$ consul kv put foo zip
$ consul kv get foo
zip
```
Consul can provide atomic key updates using a Check-And-Set operation. To perform a CAS operation, specify the `-cas` flag:
```sh
$ consul kv put -cas -modify-index=123 foo bar
Success! Data written to: foo
$ consul kv put -cas -modify-index=123 foo bar
Error! Did not write to foo: CAS failed
```
In this case, the first CAS update succeeds because the index is 123. The second
operation fails because the index is no longer 123.
## Next Steps
These are only a few examples of what the API supports. For full documentation, please
see [the /kv/ route of the HTTP API](/docs/agent/http/kv.html).
These are only a few examples of what the API supports. For the complete
documentation, please see [Consul KV HTTP API][kv-api] or
[Consul KV CLI][kv-cli] documentation.
Next, we will look at the [web UI](ui.html) options supported by Consul.
Next, we will look at the [web UI](ui.html) options supported by Consul.
Also check https://www.hashicorp.com/blog/consul-kv-cli.html for using the 'consul kv' from
the commandline.
[kv-api]: /docs/agent/http/kv.html
[kv-cli]: /docs/commands/kv.html