This includes fixes that improve gossip scalability on very large (> 10k node) clusters.
The Serf changes:
- take snapshot disk IO out of the critical path for handling messages hashicorp/serf#524
- make snapshot compaction much less aggressive - the old fixed threshold caused snapshots to be constantly compacted (synchronously with request handling) on clusters larger than about 2000 nodes! hashicorp/serf#525
Memberlist changes:
- prioritize handling alive messages over suspect/dead to improve stability, and handle queue in LIFO order to avoid acting on info that 's already stale in the queue by the time we handle it. hashicorp/memberlist#159
- limit the number of concurrent pushPull requests being handled at once to 128. In one test scenario with 10s of thousands of servers we saw channel and lock blocking cause over 3000 pushPulls at once which ballooned the memory of the server because each push pull contained a de-serialised list of all known 10k+ nodes and their tags for a total of about 60 million objects and 7GB of memory stuck. While the rest of the fixes here should prevent the same root cause from blocking in the same way, this prevents any other bug or source of contention from allowing pushPull messages to stack up and eat resources. hashicorp/memberlist#158
* Fixes TestAgent_IndexChurn
* Fixes TestPreparedQuery_Wrapper
* Increased sleep in agent_test for IndexChurn to 500ms
* Made the comment about joinWAN operation much less of a cliffhanger
Improve flaky connect/proxy Listener tests
- Add sleep to TestEchoConn to allow for Read/Write to finish before fetching data in reportStats
- Account for flakiness around interval for Gauge
- Improve debug output when dumping metrics
* Revert "BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472)"
This reverts commit cec5d72396.
* Revert "CA initialization while boostrapping and TestLeader_ChangeServerID fix. (#4493)"
This reverts commit 589b589b53.
- Improve resilience of testrpc.WaitForLeader()
- Add additionall retry to CI
- Increase "go test" timeout to 8m
- Add wait for cluster leader to several tests in the agent package
- Add retry to some tests in the api and command packages
* updates docs for agent options
trying to add a little more clarity to suggestion that folks should use
port 8302 for both LAN and WAN comms
* website: clarify language for serf wan port behavior
* Fixes the DNS recursor properly resolving the requests
* Added a test case for the recursor bug
* Refactored code && added a test case for all failing recursors
* Inner indentation moved into else if check
Adds additional 'enterprise' text underneath the 'startup' logo if the
ui is built with a CONSUL_BINARY_TYPE environment variable that doesn't
equal `oss`.
* Add some tests to check the correct GET API endpoints are called
* Refactor adapters
1. Add integration tests for `urlFor...` and majority `handleResponse` methods
2. Refactor out `handleResponse` a little more into single/batch/boolean
methods
3. Move setting of the `Datacenter` property into the `handleResponse`
method, basically the same place that the uid is being set using the dc
parsed form the URL
4. Add some Errors for if you don't pass ids to certain `urlFor` methods
Fixes: #4441
This fixes the issue with Connect Managed Proxies + ACLs being broken.
The underlying problem was that the token parsed for most http endpoints was sent untouched to the servers via the RPC request. These changes make it so that at the HTTP endpoint when parsing the token we additionally attempt to convert potential proxy tokens into regular tokens before sending to the RPC endpoint. Proxy tokens are only valid on the agent with the managed proxy so the resolution has to happen before it gets forwarded anywhere.
* Copy-and-paste Go client example
Includes Go source that runs without modification, as well as simple
instructions for compiling, running, and viewing the output in the
Consul UI.
* Remove unnecessary flags from development server example
This is a bare minimum Go example needed to store keys and values in
Consul. The `-ui` and `-server` flags aren't needed when running with
`-dev`.
* New Providers added and updated vendoring for go-discover
* Vendor.json formatted using make vendorfmt
* Docs/Agent/auto-join: Added documentation for the new providers introduced in this PR
* Updated the golang.org/x/sys/unix in the vendor directory
* Agent: TestGoDiscoverRegistration updated to reflect the addition of new providers
* Deleted terraform.tfstate from vendor.
* Deleted terraform.tfstate.backup
Deleted terraform state file artifacts from unknown runs.
* Updated x/sys/windows vendor for Windows binary compilation
* Added explainations on how having a working configuration for Prometheus
Since Prometheus escapes by default query parameters, this paragraph explains
how having `format=prometheus` to be taken into account by prometheus.
* Rephrase Prometheus notes in documentation as requested by @pearkes
Paul Banks
Siva Prasad
Armon Dadgar
Freddy
Geoffrey Grosenbach
Pierre Souchay
Jack Pearkes
freddygv
Mitchell Hashimoto
Matt Keeler
John Cowen
Matt Keeler
Jeff Escalante
Jeff Escalante
Christie Koehler
Kyle Havlovitz
Sergey Shepelev
Jason Harley