status-im
/
consul
mirror of https://github.com/status-im/consul.git
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
5,150 Commits 445 Branches 179 Tags 481 MiB
Commit Graph

1195 Commits

Author SHA1 Message Date
Sean Chittenden db72041063 Add a comment for Client serverMgr 2016-03-25 11:59:27 -07:00
Sean Chittenden 828606232e Correct a bogus goimport rewrite for tests 2016-03-23 22:35:49 -07:00
Sean Chittenden da872fee63 Test ServerManager.refreshServerRebalanceTimer 
Change the signature so it returns a value so that this can be tested externally with mock data.  See the sample table in TestServerManagerInternal_refreshServerRebalanceTimer() for the rate at which it will back off.  This function is mostly used to not cripple large clusters in the event of a partition.
 2016-03-23 22:10:50 -07:00
Sean Chittenden a63d5ab963 Add a handful more unit tests to the public interface 2016-03-23 22:10:50 -07:00
Sean Chittenden 8e3c83a258 Rename GetNumServers to NumServers() 
Matches the style of the rest of the repo
 2016-03-23 22:10:50 -07:00
Sean Chittenden 18f7befba9 Rename NewServerManger to just New 
Follow go style recommendations now that this has been refactored out of the consul package and doesn't need the qualifier in the name.
 2016-03-23 22:10:50 -07:00
Sean Chittenden e932e9a435 Rename FindHealthyServer() to FindServer() 
There is no guarantee the server coming back is healthy.  It's apt to be healthy by virtue of its place in the server list, but it's not guaranteed.
 2016-03-23 22:10:50 -07:00
Sean Chittenden 49a5a1ab84 cycleServer is a pure function, save the result 2016-03-23 22:10:50 -07:00
Sean Chittenden 94f79d2c3d Missed unit test cruft 2016-03-23 22:10:50 -07:00
Sean Chittenden bc62de541c Update comments to reflect reality 2016-03-23 22:10:50 -07:00
Sean Chittenden d2d55f4bb0 Remove additional cruft from ServerManager's channels 
No longer needed code.
 2016-03-23 22:10:50 -07:00
Sean Chittenden d13e3c18c9 Emulate a TryLock using atomic.CompareAndSwap 
Prevent possible queueing behind serverConfigLock in the event that a server fails on a busy host.
 2016-03-23 22:10:50 -07:00
Sean Chittenden 295af01680 Make use of interfaces 
Use an interface instead of serf.Serf as arg to NewServerManager.  Bonus points for improved testability.

Pointed out by: @slackpad
 2016-03-23 22:10:50 -07:00
Sean Chittenden fdbb142c3f Simplify error handling 
Rely on Serf for liveliness.  In the event of a failure, simply cycle the server to the end of the list.  If the server is unhealthy, Serf will reap the dead server.

Additional simplifications:

*) Only rebalance servers based on timers, not when a new server is readded to the cluster.
*) Back out the failure count in server_details.ServerDetails
 2016-03-23 22:10:50 -07:00
Sean Chittenden c2c73bfeab Unbreak client tests by reverting to original test 
Debugging code crept into the actual test and hung out for much longer than it should have.
 2016-03-23 22:10:50 -07:00
Sean Chittenden 0c87463b7e Introduce asynchronous management of consul server lists 
Instead of blocking the RPC call path and performing a potentially expensive calculation (including a call to `c.LANMembers()`), introduce a channel to request a rebalance.  Some events don't force a reshuffle, instead the extend the duration of the current rebalance window because the environment thrashed enough to redistribute a client's load.
 2016-03-23 22:10:50 -07:00
Sean Chittenden bad6cb8897 Comment nits 2016-03-23 22:10:50 -07:00
Sean Chittenden 74bcbc63f8 Use saveServerConfig vs atomic.Value.Store(config) 2016-03-23 22:10:50 -07:00
Sean Chittenden 7f55931d02 Commit a handful of refactoring && copy/paste-o fixes 2016-03-23 22:10:50 -07:00
Sean Chittenden e53704b032 Mutate copies of serverCfg.servers, not original 
Removing any ambiguity re: ownership of the mutated server lists is a win for maintenance and debugging.
 2016-03-23 22:10:50 -07:00
Sean Chittenden e6c27325d9 rebalanceTimer may be nil during initialization 
When first starting the server manager, it's possible that the rebalanceTimer in serverConfig will be nil, test accordingly.
 2016-03-23 22:10:50 -07:00
Sean Chittenden a7091b0837 Properly retain a pointer to the rebalanceTimer 2016-03-23 22:10:50 -07:00
Sean Chittenden 00ff8e5307 Cosmetic and various other wordsmithing cleanups 2016-03-23 22:10:50 -07:00
Sean Chittenden b4db49a62e Document the various functions and their locking 2016-03-23 22:10:50 -07:00
Sean Chittenden 9eb6481d73 Use config convenience method to get config 
'cause ELETTHECOMPILERSDOTHEWORK.  I don't need that cluttering up the subconscious with more complexity.
 2016-03-23 22:10:50 -07:00
Sean Chittenden 579e536f58 Move consul.serverConfig out of the consul package 
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
 2016-03-23 22:10:50 -07:00
Sean Chittenden c7c551dbe0 Rename serverConfigMtx to serverConfigLock 
Pointed out by: @slackpad
 2016-03-23 22:10:50 -07:00
Sean Chittenden e48b910f87 Refactor out the management of Consul servers 
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
 2016-03-23 22:10:50 -07:00
Sean Chittenden 01b637114c Move consul.serverConfig out of the consul package 
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
 2016-03-23 22:10:50 -07:00
Sean Chittenden 117c65dc55 Rename serverConfigMtx to serverConfigLock 
Pointed out by: @slackpad
 2016-03-23 22:10:32 -07:00
Sean Chittenden 0eac826573 Refactor out the management of Consul servers 
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
 2016-03-23 22:09:46 -07:00
Sean Chittenden b9e5588620 Move consul.serverConfig out of the consul package 
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
 2016-03-23 22:05:29 -07:00
Sean Chittenden a482eaef70 Rename serverConfigMtx to serverConfigLock 
Pointed out by: @slackpad
 2016-03-23 22:05:05 -07:00
Sean Chittenden 9b8767aa67 Refactor out the management of Consul servers 
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
 2016-03-23 22:03:20 -07:00
Sean Chittenden 075d1b628f Commit miss re: consuls variable rename 2016-03-23 16:24:29 -07:00
Sean Chittenden 2ca4cc58ce Move consul.serverConfig out of the consul package 
Relocated to its own package, server_manager.  This now greatly simplifies the RPC() call path and appropriately hides the locking behind the package boundary.  More work is needed to be done here
 2016-03-23 16:16:22 -07:00
Sean Chittenden 0925b26250 Refactor consul.serverParts into server_details.ServerDetails 
This may be short-lived, but it also seems like this is going to lead us down a path where ServerDetails is going to evolve into a more powerful package that will encapsulate more behavior behind a coherent API.
 2016-03-23 16:15:47 -07:00
Sean Chittenden 5be956c310 Rename serverConfigMtx to serverConfigLock 
Pointed out by: @slackpad
 2016-03-23 16:15:47 -07:00
Sean Chittenden b1e392405c Handle the case where there are no healthy servers 
Pointed out by: @slackpad
 2016-03-23 16:15:47 -07:00
Sean Chittenden d4ca349e21 Refactor out the management of Consul servers 
Move the management of c.consulServers (fka c.consuls) into consul/server_manager.go.

This commit brings in a background task that proactively manages the server list and:

*) reshuffles the list
*) manages the timer out of the RPC() path
*) uses atomics to detect a server has failed

This is a WIP, more work in testing needs to be completed.
 2016-03-23 16:15:47 -07:00
Sean Chittenden 7b308d8d7e Add a flag to denote that a server is disabled 
A server is not normally disabled, but in the event of an RPC error, we want to mark a server as down to allow for fast failover to a different server.  This value must be an int in order to support atomic operations.

Additionally, this is the preliminary work required to bring up a server in a disabled state.  RPC health checks in the future could mark the server as alive, thereby creating an organic "slow start" feature for Consul.
 2016-03-23 16:14:59 -07:00
Sean Chittenden 6af781d9d5 Rename `lastServer` to `preferredServer` 
Expanding the domain of lastServer beyond RPC() changes the meaning of this variable.  Rename accordingly to match the intent coming in a subsequent commit: a background thread will be in charge of rotating preferredServer.
 2016-03-23 16:14:59 -07:00
Sean Chittenden f6ffbf4e96 Warn if serf events have queued up past 80% of the limit 
It is theoretically possible that the number of queued serf events can back up.  If this happens, emit a warning message if there are more than 200 events in queue.

Most notably, this can happen if `c.consulServerLock` is held for an "extended period of time".  The probability of anyone ever seeing this log message is hopefully low to nonexistent, but if it happens, the warning message indicating a large number of serf events fired while a lock was held is likely to be helpful (vs serf mysteriously blocking when attempting to add an event to a channel).
 2016-03-23 16:14:11 -07:00
Sean Chittenden 54016f5276 Commit miss re: consuls variable rename 2016-03-23 16:13:49 -07:00
Sean Chittenden f9aa968bf4 Remove lastRPCTime 
This mechanism isn't going to provide much value in the future.  Preemptively reduce the complexity of future work.
 2016-03-23 16:13:49 -07:00
Sean Chittenden cc86eb0a1a Rename c.consuls to c.consulServers 
Prep for breaking out maintenance of consuls into a new goroutine.
 2016-03-23 16:10:27 -07:00
Sean Chittenden 146c5b0a59 Use `rand.Int31n()` to get power of two optimization 
In cases where i+1 is a power of two, skip one modulo operation.
 2016-03-23 16:00:39 -07:00
James Phillips 8a9ad3811b Gets rid of flaky sort check. 
If we get a coordinate then this test will fail, so we only check the
first item in the list, which is deterministic.
 2016-03-21 17:30:05 -07:00
James Phillips fffa8d5061 Increases timeouts for coordinate tests. 
We take the interval and add the random stagger to it, so 2X is cutting it
too close and the unit tests are often flaky.
 2016-03-21 16:44:35 -07:00
James Phillips b6cd4318d6 Merge pull request #1851 from hashicorp/f-ipv6-bind 
Allow [::] as a bind address (binds to first public IPv6 address)
 2016-03-19 16:16:19 -07:00
James Phillips 99eb629c8f Adds more specific checks for ipv6 addresses. 2016-03-19 16:14:45 -07:00
James Phillips 3c883951d7 Removes leader from members and changes name since it's an address. 2016-03-18 17:07:11 -07:00
Sergey Romanov 93c8b496e5 #735 add information about leader to consul members 2016-03-18 17:05:40 -07:00
Wim b5d45322b4 Allow [::] as a bind address (binds to first public IPv6 address) 2016-03-18 23:59:44 +01:00
Calvin Leung Huang 4bd5523276 Obfuscate token for lookupACL error 2016-03-15 17:16:25 -04:00
James Phillips cb9c908a99 Hardens the match interoplator against negative arguments. 2016-03-07 13:32:32 -08:00
James Phillips 1cad6b9e0f Adds a comment about the embedded struct. 2016-03-07 10:45:39 -08:00
James Phillips eb7004f2b8 Renames "debug" endpoint and structures to "explain". 2016-03-07 10:45:39 -08:00
James Phillips d7288e3a5e Adds a prepared query debug endpoint. 2016-03-07 10:45:39 -08:00
James Phillips c7ee82c67f Applies prefix ACL to a catch-all template as a special case. 2016-03-07 10:45:39 -08:00
James Phillips 79eccf2c66 Adds a test for the custom prepared query template indexer. 2016-03-07 10:45:39 -08:00
James Phillips 897ab0d5c7 Adds core query template tests to the state store. 2016-03-07 10:45:39 -08:00
James Phillips 328d138466 Adds in basic query template lookups and vendors newly-updated memdb as well as improved iradix tree. 2016-03-07 10:45:39 -08:00
James Phillips 07514214e1 Adds tests for the low-level template functions. 2016-03-07 10:45:39 -08:00
James Phillips e3827923b8 Adds tests for the string visitor. 2016-03-07 10:45:39 -08:00
James Phillips 799339acb5 Factors rendering down into the resolve function. 2016-03-07 10:45:39 -08:00
James Phillips 6ed64e7f05 Splits walk functions out from the rest of the template code. 2016-03-07 10:45:39 -08:00
James Phillips 998b691878 Integrates templates into state store and endpoint (sans tests). 2016-03-07 10:45:39 -08:00
James Phillips c816f79bf8 Wraps the prepared query to also store the compiled template. 2016-03-07 10:45:39 -08:00
James Phillips 331f1f5b8b Adds basic query template compiler and renderer. 2016-03-07 10:45:39 -08:00
Mike Cowgill 5435055c5f one line schema change to not allow missing for sessions Table node index, Fixes #1774 2016-03-02 21:19:53 -08:00
James Phillips 90898dff98 Adds missing token redact in the GET path. 2016-02-26 15:59:00 -08:00
James Phillips 213026b033 Merge pull request #1757 from hashicorp/f-revert-1667 
Reverts server connection rebalancing changes from #1667
 2016-02-24 18:07:13 -08:00
James Phillips 7d392118d2 Adds a check for users re-submitting the redacted token. 2016-02-24 17:35:26 -08:00
James Phillips 483898abe5 Renames "prepared_query" ACL policy to "query". 2016-02-24 17:02:06 -08:00
James Phillips 87ceb2f3de Changes to more idiomatic "ok" pattern for prefix getter. 2016-02-24 16:26:43 -08:00
James Phillips e283f9512e Renames a unit test. 2016-02-24 16:17:20 -08:00
James Phillips ff25d033a6 Revert "Merge pull request #1667 from hashicorp/b-redistribute-clients" 
This reverts commit 8f30dea420, reversing
changes made to eb27a02956.
 2016-02-24 15:38:03 -08:00
James Phillips 899dcfe053 Completes switch of prepared_query ACLs to govern query names. 2016-02-24 01:26:16 -08:00
James Phillips 67de77482e Creates new "prepared-query" ACL type and new token capture behavior. 
Prior to this change, prepared queries had the following behavior for
ACLs, which will need to change to support templates:

1. A management token, or a token with read access to the service being
   queried needed to be provided in order to create a prepared query.

2. The token used to create the prepared query was stored with the query
   in the state store and used to execute the query.

3. A management token, or the token used to create the query needed to be
   supplied to perform and CRUD operations on an existing prepared query.

This was pretty subtle and complicated behavior, and won't work for
templates since the service name is computed at execution time. To solve
this, we introduce a new "prepared-query" ACL type, where the prefix
applies to the query name for static prepared query types and to the
prefix for template prepared query types.

With this change, the new behavior is:

1. A management token, or a token with "prepared-query" write access to
   the query name or (soon) the given template prefix is required to do
   any CRUD operations on a prepared query, or to list prepared queries
   (the list is filtered by this ACL).

2. You will no longer need a management token to list prepared queries,
   but you will only be able to see prepared queries that you have access
   to (you get an empty list instead of permission denied).

3. When listing or getting a query, because it was easy to capture
   management tokens given the past behavior, this will always blank out
   the "Token" field (replacing the contents as <hidden>) for all tokens
   unless a management token is supplied. Going forward, we should
   discourage people from binding tokens for execution unless strictly
   necessary.

4. No token will be captured by default when a prepared query is created.
   If the user wishes to supply an execution token then can pass it in via
   the "Token" field in the prepared query definition. Otherwise, this
   field will default to empty.

5. At execution time, we will use the captured token if it exists with the
   prepared query definition, otherwise we will use the token that's passed
   in with the request, just like we do for other RPCs (or you can use the
   agent's configured token for DNS).

6. Prepared queries with no name (accessible only by ID) will not require
   ACLs to create or modify (execution time will depend on the service ACL
   configuration). Our argument here is that these are designed to be
   ephemeral and the IDs are as good as an ACL. Management tokens will be
   able to list all of these.

These changes enable templates, but also enable delegation of authority to
manage the prepared query namespace.
 2016-02-23 17:12:43 -08:00
James Phillips 778a26efaf Adds a test for node registration and tagged addresses. 2016-02-07 13:15:22 -08:00
James Phillips 4be2ab1a75 Moves tagged wan address to be managed by anti-entropy, not serf. 2016-02-07 13:12:42 -08:00
James Phillips 33462ebea9 Adds an FSM persist and restore test for tagged addresses. 2016-02-07 11:36:39 -08:00
James Phillips c60a526fde Sets up config for more address tags down the road, renames struct members. 2016-02-07 10:37:34 -08:00
Evan Gilman de8fd561d0 Use a map for additional node addresses 2016-02-06 23:01:45 -08:00
Evan Gilman 069a28b3c0 Use idiomatic name for wan_addr serf tag 2016-02-06 23:01:45 -08:00
James Phillips ed8a71efd7 Store WanAddress during Service/Check sync 2016-02-06 23:01:45 -08:00
Evan Gilman 9300a13643 Store WanAddress during node registration 2016-02-06 23:01:45 -08:00
Evan Gilman 90aafbbdb6 Store WanAddress on Node 2016-02-06 23:01:45 -08:00
Sean Chittenden 1f725e2d05 Use the server's address in debug logging, not the c.lastServer, which may be nil 2016-02-02 15:51:28 -08:00
Sean Chittenden 1c9d74a337 Remove unnecessary check, test was moved further up in scope 2016-02-02 11:13:58 -08:00
Sean Chittenden 1005b91c87 Use panic instead of returning a sentinel UUID values in unit tests 2016-02-01 23:15:19 -08:00
Sean Chittenden c7e58734ed Continually rebalance client connections 
Introduce a low-level background connection expiration mechanism wherein connections will be recycled periodically based on the size and health of the cluster.

For the vast majority of consul users, this will mean an average connection age of 150s.  For 10K node clusters it will take ~3min for clusters to rebalance their connections.  In the pathological case for a 100K cluster where 99K clients are in the minority talking to 1x server it will take ~26min to rebalance all connections.

It's possibe for clients recovering from a parititon to become fixated on a single server until the server or agent is restarted.  This is of particular interest to long-running environments with stable agents, where `allow_stale` is true, and partitions occur periodically.
 2016-01-30 17:13:50 -08:00
Sean Chittenden 0c83b1b692 Use rand.Int31n() vs unconditionally using modulus 2016-01-30 15:47:58 -08:00
Sean Chittenden 7fb0045bbe Merge branch 'f-consul-lib' of ssh://github.com/hashicorp/consul into b-redistribute-clients 2016-01-30 15:40:54 -08:00
Sean Chittenden c4f7b4a13e Rename clientRPCCache to clientRPCConnMaxIdle, change value 
Increase the max idle time for agents talking to servers from 30s to 127s in order to allow for the reuse of connections that are being initiated by cron.

127s was chosen as the first prime above 120s (arbitrarily chose to use a prime) with the intent of reusing connections who are used by once-a-minute cron(8) jobs *and* who use a 60s jitter window (e.g. in vixie cron job execution can drift by up to 59s per job, or 119s for a once-a-minute cron job).
 2016-01-30 15:27:46 -08:00
Sean Chittenden b391b075bd Reuse the results from gettimeofday(2)... 
Inside of a single RPC call, reuse time.Now().
 2016-01-30 14:39:17 -08:00
Sean Chittenden 7af6a94edb Factor out duplicate functions into a lib package 
Consolidate code duplication and tests into a single lib package.  Most of these functions were from various **/util.go functions that couldn't be imported due to cyclic imports.  The consul/lib package is intended to be a terminal node in an import DAG and a place to stash various consul-only helper functions.  Pulled in hashicorp/go-uuid instead of consolidating UUID access.
 2016-01-29 16:57:45 -08:00
James Phillips 01da5a2248 Prevents watches from being orphaned when KVS blocking queries loop. 2016-01-20 07:18:47 -08:00
James Phillips 94d3f881fe Merge pull request #948 from hashicorp/iface-down-fix 
Don't try to bind on address from inactive interface
 2016-01-14 17:00:54 -08:00
Ryan Uber 2bcaa8ac38 consul: address comments 2016-01-05 09:45:36 -08:00
Ryan Uber 90566d4bde consul: disable serf snapshots in dev mode 2015-12-26 20:39:49 -05:00
Ryan Uber afafae53fd consul: dev mode works 2015-12-26 20:19:36 -05:00
James Phillips 18148b9af8 Cleans up some small `go vet` findings. 2015-12-18 22:14:48 -08:00
James Phillips 6dc151b1b9 Adds source address logging on RPC errors. 2015-12-16 14:03:03 -08:00
James Phillips e25337866c Adds a check to make sure query names can't be registered twice. 2015-12-02 09:04:51 -08:00
Armon Dadgar 847343d801 consul: shrink yamux recv buffer on idle streams 2015-11-27 17:20:57 -08:00
James Phillips e3ef204ac7 Makes all the query ops the correct type. 2015-11-17 09:27:10 -08:00
James Phillips 533e6fc89e Returns a zero index for a lookup error case. 2015-11-17 09:25:20 -08:00
James Phillips 1059a8b3a3 Removes a useless empty import and fixes some stale comments. 2015-11-17 08:29:20 -08:00
James Phillips 06b918e46d Makes UUID regex case-insensitive. 2015-11-16 22:57:47 -08:00
James Phillips e1ce1a34b0 Moves conversion of nil slices up to HTTP layer for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips b8ddb21978 Adds a paranoia set of the nodes slice to nil. 2015-11-15 17:06:00 -08:00
James Phillips 46d5afa574 Adds a test to ensure we don't return a nil slice. 2015-11-15 17:06:00 -08:00
James Phillips f60fc872d1 Gets rid of some unused constants. 2015-11-15 17:06:00 -08:00
James Phillips 67fd4fa78d Returns a 404 from a get or execute of a nonexistent query. 2015-11-15 17:06:00 -08:00
James Phillips e9480ecb02 Plumbs the service name back and uses agent-specific TTL settings as a fallback. 2015-11-15 17:06:00 -08:00
James Phillips 81b43135f9 Adds unit tests for prepared queries and DNS, using existing tests for equivalence. 2015-11-15 17:06:00 -08:00
James Phillips 6634cd6567 Adds query metadata to prepared query execute response. 2015-11-15 17:06:00 -08:00
James Phillips c955799baf Makes an empty prepared query list an empty slice, not a nil one. 2015-11-15 17:06:00 -08:00
James Phillips 34b685cb4c Adds a unit test for the new RTT getDatacentersByDistance fn. 2015-11-15 17:06:00 -08:00
James Phillips 8e1bea0192 Completes FSM support for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 57be55103c Adds an HTTP endpoint for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 5d06a87d82 Adds an RPC endpoint injection method for testing. 2015-11-15 17:06:00 -08:00
James Phillips 7af41edf52 Changes Lookup to Get since we don't need it (only Execute does). 2015-11-15 17:06:00 -08:00
James Phillips a57d642fa0 Always increments the failovers counter, even for error-ed DCs. 2015-11-15 17:06:00 -08:00
James Phillips d06e2a535d Adds test for remote datacenter selection and query logic. 2015-11-15 17:06:00 -08:00
James Phillips bbc5185000 Adds a test for the server wrapper. 2015-11-15 17:06:00 -08:00
James Phillips eefdb56d1e Adds tag filter tests. 2015-11-15 17:06:00 -08:00
James Phillips 14170535e7 Adds execute tests for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 30a18220af Adds status information about failovers to query results. 2015-11-15 17:06:00 -08:00
James Phillips 86ead892ab Removes unused ACL filter. 2015-11-15 17:06:00 -08:00
James Phillips a9e9d5e311 Adds execute leader forward test for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips 7ded6c7a4a Adds a leader forwarding case for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips fa414a2092 Adds tests for query lookup and list endpoints. 2015-11-15 17:06:00 -08:00
James Phillips 519666a97c Adds query parsing unit tests. 2015-11-15 17:06:00 -08:00
James Phillips d4d866c6d5 Adds ACL cases for apply. 2015-11-15 17:06:00 -08:00
James Phillips 8222d3f462 Completes non-ACL version of apply test. 2015-11-15 17:06:00 -08:00
James Phillips 333da2a96c Adds lookup and list endpoints and basic end-to-end apply test. 2015-11-15 17:06:00 -08:00
James Phillips 58bb6e8ba4 Checks for valid UUIDs before calling in to index function. 2015-11-15 17:06:00 -08:00
James Phillips 0bd7e82686 Clarifies comment about name vs. ID. 2015-11-15 17:06:00 -08:00
James Phillips 666619dfc9 Skips unknown DCs during queries and chugs along in the face of errors. 2015-11-15 17:06:00 -08:00
James Phillips 2f34b51650 Moves sort to a query-time decision and adds back the limit. 2015-11-15 17:06:00 -08:00
James Phillips 7ca3f0a466 Adds an explicit ACL check that will fail vs. trying other DCs. 2015-11-15 17:06:00 -08:00
James Phillips c41a3d6c8d Changes "not" prefix from "~" to "!". 2015-11-15 17:06:00 -08:00
James Phillips 81bb39751a Adds prefix "prepared" to everything prepared query-related. 2015-11-15 17:06:00 -08:00
James Phillips b736bc4e68 Adds basic structure for prepared queries (needs tests). 2015-11-15 17:06:00 -08:00
James Phillips 1d1865ddff Factors code for pulling the sorted list of DCs into a common place. 2015-11-15 17:06:00 -08:00
James Phillips 1f87480e54 Adds a better shuffle test (similar to DNS). 2015-11-15 17:06:00 -08:00
James Phillips 7babcefc59 Changes structs and state store for prepared queries. 2015-11-15 17:06:00 -08:00
James Phillips ff351b289a Merge pull request #1413 from hashicorp/b-coord-raft-errors 
Adds a check for in-band error returns in the coordinate RaftApply.
 2015-11-15 17:05:19 -08:00
James Phillips f1f9d30ade Extends the session TTL max to 24 hours, and adds a warning to the docs. 2015-11-15 16:51:00 -08:00
James Phillips 9972eb2498 Adds a check for in-band error returns in the coordinate RaftApply. 2015-11-15 16:50:46 -08:00
James Phillips c1d338550f Changes batch update failure to a WARN since it's nbd. 2015-10-29 09:26:24 -07:00
James Phillips d06fc28a0a Prevents agents from considering Raft information when doing sync checks. 2015-10-28 14:32:00 -07:00
James Phillips 19c3e262a6 Fixes a bad error message. 2015-10-28 12:40:47 -07:00
James Phillips 9f85f7ee1a Adds a check for the minimum version as well. 2015-10-27 15:56:36 -07:00
James Phillips 26095faae0 Sets the ignore flag on coordinate update Raft log entries. 2015-10-27 14:44:34 -07:00
James Phillips c0bd639662 Makes the version upshift code look at the correct version field. 2015-10-27 14:44:34 -07:00
James Phillips ecd3a1d1d2 Completes rebase of network coordinates to new memdb. 2015-10-23 15:23:01 -07:00
James Phillips de01f96d38 Fixes configs now that Serf always caches coordinates. 2015-10-23 15:23:01 -07:00
James Phillips 660da92152 Makes the default protocol 2 and lets 3 interoperate with 2. 2015-10-23 15:23:01 -07:00
James Phillips 80d5a3029f Zeroes out the height when testing exact distances. 2015-10-23 15:23:01 -07:00
James Phillips 384d99677d Cleans up after latest rebase. 2015-10-23 15:23:01 -07:00
James Phillips 6289764ea2 Moves sorting up into coordinate endpoint HTTP handlers. 2015-10-23 15:23:01 -07:00
James Phillips 33e3505aea Adds endpoints for raw network coordinates. 2015-10-23 15:23:01 -07:00
James Phillips 843613143b Fixes bad name for DC forwarding of Coordinate.Get. 2015-10-23 15:23:01 -07:00
James Phillips 9caa5b3653 Adds distance sorting to health endpoint. Cleans up unit tests. 2015-10-23 15:23:01 -07:00
James Phillips 497f6782af Switches to the median over all DC nodes with known coordinates. 2015-10-23 15:23:01 -07:00
James Phillips 6d845c7d0d Adds explicit check for empty node in source parameter. 2015-10-23 15:23:01 -07:00
James Phillips 78b2c2d7ac Moves disable checks down into the sort routine. 2015-10-23 15:23:01 -07:00
James Phillips 54ef97b268 Adds tests for HTTP interface. Removes a stray mark. 2015-10-23 15:23:01 -07:00
James Phillips 36c78f5042 Adds sort of DCs in catalog queries based on RTT. Cleans up. 
* Makes the catalog endpoint respect disabling coordinates for all
  RTT-sorting query types.
 2015-10-23 15:23:01 -07:00
James Phillips 89c7203f31 Adds coordinate sorting support to catalog queries for nodes and service nodes. 2015-10-23 15:23:01 -07:00
James Phillips ad65d953f6 Scales coordinate sends to hit a fixed aggregate rate across the cluster. 2015-10-23 15:23:01 -07:00
James Phillips 66a3d29743 Simplifies the batching function and adds some comments. 2015-10-23 15:23:01 -07:00
James Phillips e094f5a61d Adds snapshot save and restore of coordinates. 2015-10-23 15:23:01 -07:00
James Phillips 5f754c4a87 Does some small cleanups based on PR feedback. 
* Holds coordinate updates in map and gets rid of the update channel.
* Cleans up config variables a bit.
 2015-10-23 15:23:01 -07:00
James Phillips 7e6d52109b Hardens Consul from bad coordinates from other nodes. 2015-10-23 15:23:01 -07:00
James Phillips 2cee9f7e2f Takes the node name out of the coordinate get call. 2015-10-23 15:23:01 -07:00
James Phillips d12aa2ffab Moves batching down into the state store and changes it to fail-fast. 
* A batch of updates is done all in a single transaction.
* We no longer need to get an update to kick things, there's a periodic flush.
* If incoming updates overwhelm the configured flush rate they will be dumped with an error.
 2015-10-23 15:23:01 -07:00
James Phillips b9d5fb0f90 Flips the sense of the coordinate enable option. 2015-10-23 15:23:01 -07:00
James Phillips 92567841d6 Removes one more WAN leftover. 2015-10-23 15:23:01 -07:00
James Phillips 86b112fe31 Does a clean up pass on the Consul side. 2015-10-23 15:23:01 -07:00
James Phillips 01d2452ea3 Merges config changes after rebase. 2015-10-23 15:23:01 -07:00
Derek Chiang 7255ddd086 Address comments 2015-10-23 15:23:01 -07:00
Derek Chiang 2f11db15aa Add test for the GetWAN endpoint 2015-10-23 15:23:01 -07:00
Derek Chiang 1929b1dabf Add an endpoint for getting WAN coordinates 2015-10-23 15:23:01 -07:00
Derek Chiang 69003310ca Fix tests 2015-10-23 15:23:01 -07:00
Derek Chiang 54a2815d9c Address comments 2015-10-23 15:23:01 -07:00
Derek Chiang f8d81092c9 Add state store tests 2015-10-23 15:23:01 -07:00
Derek Chiang f657caf1dd Add more tests 2015-10-23 15:23:01 -07:00
Derek Chiang f144d17b1c Address comments 2015-10-23 15:23:01 -07:00
Derek Chiang bf5cb7522f Use IndexedCoordinate instead 2015-10-23 15:23:01 -07:00
Derek Chiang d8dd8d77d4 Improve a test 2015-10-23 15:23:01 -07:00
Derek Chiang 98d87b5dd5 Complete logic for sending coordinates 2015-10-23 15:23:01 -07:00
Derek Chiang 8a0bb40bba Fix tests 2015-10-23 15:23:01 -07:00
Derek Chiang 9113b26286 Some fixes 2015-10-23 15:23:01 -07:00
Derek Chiang b87f122a5b Change GET API a little bit 2015-10-23 15:23:01 -07:00
Derek Chiang a9ea503c69 Adding tests and stuff 2015-10-23 15:23:01 -07:00
Derek Chiang 13f6cd376f Start adding stuff 2015-10-23 15:23:01 -07:00
James Phillips 158eabdd6f Cleans up a little whitespace with go fmt. 2015-10-20 18:41:05 -07:00
James Phillips 785bf6678d Gets rid of error prefixing in leader.go. 2015-10-20 13:37:11 -07:00
James Phillips f3a95bf9fe Puts all restore operations into a single transaction and optimizes watches. 2015-10-19 23:06:59 -07:00
James Phillips 52c373bb65 Gets rid of unique constraint on sessions index in session_checks. 2015-10-19 17:41:50 -07:00
James Phillips ee52f5ebce Switches sessions over to UUIDFieldIndex. 2015-10-19 17:09:02 -07:00
James Phillips 7d1179c1eb Adds a special case for fill KVS listings to avoid a tombstone scan. 2015-10-19 16:22:27 -07:00
James Phillips 3782fc53cb Makes the iterator naming more consistent. 2015-10-19 15:51:11 -07:00
James Phillips ffe531c55f Converts sessions and ACLs over to iterators. 2015-10-19 14:56:22 -07:00
James Phillips 8371c87fd0 Converts KVS snapshot over to iterator. 2015-10-19 14:07:57 -07:00
James Phillips 503c552d28 Converts nodes, services, checks to iterators duing dumps; fixes tag drift bug. 
Realized that the conversions ServiceNode <-> NodeService were incomplete in a
few places so centralized those and added some tests.
 2015-10-19 13:55:35 -07:00
James Phillips 4fb8d1078e Gets rid of non-idomatic "state_store" alias in FSM. 2015-10-16 14:07:48 -07:00
James Phillips 8d444f2eed Ports a couple of new RPC calls to the updated codec mechanism. 2015-10-15 15:09:13 -07:00
James Phillips 655967d0f6 Runs go fmt after latest changes. 2015-10-15 14:59:09 -07:00
James Phillips ef5233165b Adds unit tests for new structs clone functions. 2015-10-15 14:59:09 -07:00
James Phillips a1cb9b8e62 Adds benchmarks back in to the state store. 2015-10-15 14:59:09 -07:00
James Phillips fc541fadcf Removes a todo that's no longer needed. 2015-10-15 14:59:09 -07:00
James Phillips be9f071488 Adds unit tests for the graveyard. 2015-10-15 14:59:09 -07:00
James Phillips fab8672543 Does some go fmt after latest round of changes. 2015-10-15 14:59:09 -07:00
James Phillips 57c102a7f1 Adds a delay test. 2015-10-15 14:59:09 -07:00
James Phillips 6f7f163a76 Adds watch unit tests and does some related watch cleanup. 2015-10-15 14:59:09 -07:00
James Phillips d29673ead7 Adds tests for GC. 2015-10-15 14:59:09 -07:00
James Phillips d2f4a5fe83 Adds clone for service nodes so we don't twiddle the database's object. 2015-10-15 14:59:09 -07:00
James Phillips b6f9aee847 Fixes remaining non-KV index calclulations and adds a general getWatchTables thing. 2015-10-15 14:59:09 -07:00
James Phillips 181c216f53 Fixes index management for KVS. 2015-10-15 14:59:09 -07:00
James Phillips ca3a84edf8 Makes session invalidate loops use a separate slice to protect the iterator. 2015-10-15 14:59:09 -07:00
James Phillips 834c6c1cb4 Makes all delete loops use a separate slice to protect the iterator. 2015-10-15 14:59:09 -07:00
James Phillips 682b0113b8 Cleans up some go vet warnings. 2015-10-15 14:59:09 -07:00
James Phillips cc6a7a2174 Gets rid of todo that we discussed is ok. 2015-10-15 14:59:09 -07:00
James Phillips 51600fab08 Gets rid of the transitional "New" suffix on state store and RPC. 2015-10-15 14:59:09 -07:00
James Phillips 4ee43e90b7 Deletes the old state store and all its accoutrements. 2015-10-15 14:59:09 -07:00
James Phillips 6ba70be33c Nukes old state store's connection to FSM and RPC. 2015-10-15 14:59:09 -07:00
James Phillips 7729b66099 Integrates session TTL tests with new state store. 2015-10-15 14:59:09 -07:00
James Phillips c83a9e043a Integrates new state store into internal endpoint. 2015-10-15 14:59:09 -07:00
James Phillips 0959b87548 Abstracts the table names away from the RPC call sites. 2015-10-15 14:59:09 -07:00
James Phillips b82d49240b Integrates new state store into session endpoint; returns table index always. 2015-10-15 14:59:09 -07:00
James Phillips 5b6502c1ea Integrates new state store into leader and catalog/health endpoints. 2015-10-15 14:59:09 -07:00
James Phillips 41338c9018 Integrates KVS endopint with new state store (changes KVSList to match old behavior). 2015-10-15 14:59:09 -07:00
James Phillips de00a2f81b Cuts FSM unit tests over to new state store. 2015-10-15 14:59:09 -07:00
James Phillips c791f2a709 Adds twiddling of the real state store in snapshot tests. 2015-10-15 14:59:09 -07:00
James Phillips 91ff5254dd Ports over session invalidation tests (and fixes some bugs). 2015-10-15 14:59:09 -07:00
James Phillips c3513b1661 Adds a note about updating sessions after they are created. 2015-10-15 14:59:09 -07:00
James Phillips 87bb81ba4c Beefs up node and service watch tests for multi-table triggers. 2015-10-15 14:59:09 -07:00
James Phillips bde2495a14 Adds session snapshot/restore and basic watch tests (and fixes some bugs). 2015-10-15 14:59:09 -07:00
James Phillips 4afd9a8599 Adds tombstone tests and gets rid of unused logger. 2015-10-15 14:59:09 -07:00
James Phillips ad246e06a5 Makes sure we don't create a full table watch for tombstones. 2015-10-15 14:59:09 -07:00
James Phillips 930780efd9 Adds snapshot/restore and watch tests for KVS. 2015-10-15 14:59:09 -07:00
James Phillips 18d60f95c5 Allows lock holder to re-lock and set a KV, adds tests for corner cases around sessions. 2015-10-15 14:59:09 -07:00
James Phillips 75f9cd5cc3 Adds kvs lock/unlock tests. 2015-10-15 14:59:09 -07:00
James Phillips 72f15ceac0 Adds snap and watch tests for nodes, services, and checks. 2015-10-15 14:59:09 -07:00
James Phillips 005a7e0ab1 Adds node snap and watch tests as well as a general watch test. 2015-10-15 14:59:09 -07:00
James Phillips c8d0d09087 De-generalizes graveyard since that ended up as a YAGNI (only useful for KV). 2015-10-15 14:59:09 -07:00
James Phillips 537fd67464 Cleans up unit tests for consistency. 2015-10-15 14:59:09 -07:00
James Phillips f1ee05eba5 Adds ensure registration unit test (and fixes bugs). 2015-10-15 14:59:09 -07:00
James Phillips e339ebf86e Adds reap tombstone test (and fixes bugs). 2015-10-15 14:59:09 -07:00
James Phillips bc34ae2f5c Adds ACL snapshot/restore test. 2015-10-15 14:59:09 -07:00
James Phillips 21bc8e0996 Adds a watch tester helper that helps cut the cruft. 2015-10-15 14:59:09 -07:00
James Phillips 30736bae5a Completes state store for KV, sessions, tombstones, and nodes/services/checks (needs tests and integration). 2015-10-15 14:59:09 -07:00
James Phillips 009fd7d9f5 Integrates new state store for ACLs. 2015-10-15 14:59:09 -07:00
James Phillips d57431e300 Gets new structs changes to compile, adds some corner case handling and extra unit tests. 2015-10-15 14:59:09 -07:00
Ryan Uber 7bf7ba6099 consul/state: list acls 2015-10-15 14:59:09 -07:00
Ryan Uber 92191298d0 consul/state: implement acl delete 2015-10-15 14:59:09 -07:00
Ryan Uber 16188e7604 consul/state: basic acl set/get/delete 2015-10-15 14:59:09 -07:00
Ryan Uber 391d4eed57 consul/state: basic session destroy works 2015-10-15 14:59:09 -07:00
Ryan Uber 8fa82612d2 consul/state: more tests 2015-10-15 14:59:09 -07:00
Ryan Uber 876aa667ba consul/state: refactor some tests 2015-10-15 14:59:09 -07:00
Ryan Uber a93e3410b9 consul/state: session lookup by node id works 2015-10-15 14:59:09 -07:00
Ryan Uber bd0de2c1e4 consul/state: add session list method 2015-10-15 14:59:09 -07:00
Ryan Uber a4c202aa90 consul/state: adding session registration 2015-10-15 14:59:09 -07:00
Ryan Uber b0856c2215 consul/state: adding tree delete for kvs store 2015-10-15 14:59:09 -07:00
Ryan Uber 291fbe02ba consul/state: list keys from the kv with a prefix/separator 2015-10-15 14:59:09 -07:00
Ryan Uber 8b29bfa303 consul/state: fix for maxIndex and better tests 2015-10-15 14:59:09 -07:00
Ryan Uber a0dc2ded8d consul/state: add CAS method for kv set 2015-10-15 14:59:09 -07:00
Ryan Uber 8a70ba2cc5 consul/state: initial pass at CAS delete operation for kvs 2015-10-15 14:59:09 -07:00
Ryan Uber 4ba89adb7d consul/state: adding KVSList for listing a given prefix 2015-10-15 14:59:09 -07:00
Ryan Uber a0fd9feda3 consul/state: adding shallow delete for kvs store 2015-10-15 14:59:09 -07:00
Ryan Uber 08d41224a3 consul/state: basic k/v operations 2015-10-15 14:59:09 -07:00
Ryan Uber f05a322dc7 consul/state: adding node dump methods 2015-10-15 14:59:09 -07:00
Ryan Uber 3b36744277 consul/state: testing service registration update 2015-10-15 14:59:09 -07:00
Ryan Uber 26f717f215 consul/state: fetch node/check sets by service ID 2015-10-15 14:59:09 -07:00
Ryan Uber f54a96f56b consul/state: filter checks by state 2015-10-15 14:59:09 -07:00
Ryan Uber 2cde299be9 consul/state: support check lookups by service name 2015-10-15 14:59:09 -07:00
Ryan Uber a4a73c3904 consul/state: return highest index for queries with compound results 2015-10-15 14:59:09 -07:00
Ryan Uber 9f281cbf80 consul/state: cleanup 2015-10-15 14:59:09 -07:00
Ryan Uber e148bb55d8 consul/state: better tests for index table updates 2015-10-15 14:59:09 -07:00
Ryan Uber fa8e015219 consul/state: test helpers 2015-10-15 14:59:09 -07:00
Ryan Uber 04d7e5849a consul/state: node checks are removed with their associated nodes 2015-10-15 14:59:09 -07:00
Ryan Uber aa14ca3a7c consul/state: services are removed with their associated nodes 2015-10-15 14:59:09 -07:00
Ryan Uber 0809a7e825 consul/state: remove checks during service deregistration 2015-10-15 14:59:09 -07:00
Ryan Uber b0132b140c consul/state: add check deletion method 2015-10-15 14:59:09 -07:00
Ryan Uber 6ebed234bb consul/state: set index if we have an existing health check 2015-10-15 14:59:09 -07:00
Ryan Uber 8c6f40f299 consul/state: negative tests 2015-10-15 14:59:09 -07:00
Ryan Uber 77f14af977 consul/state: basic health check retrieval works 2015-10-15 14:59:09 -07:00
Ryan Uber cf569f4848 consul/state: persisting health checks works 2015-10-15 14:59:09 -07:00
Ryan Uber 1d4a6ea790 consul/state: test for index modification during deletes 2015-10-15 14:59:09 -07:00
Ryan Uber 8a8aef0503 consul/state: add service delete functions 2015-10-15 14:59:09 -07:00
Ryan Uber 8c38210559 consul/state: working on node deletion 2015-10-15 14:59:09 -07:00
Ryan Uber 66b3250e55 consul/state: add function for returning all nodes 2015-10-15 14:59:09 -07:00
Ryan Uber 8671c5f80b consul/state: track highest index when querying services 2015-10-15 14:59:09 -07:00
Ryan Uber a52ed3c35b consul/state: querying node services works 2015-10-15 14:59:09 -07:00