This includes fixes that improve gossip scalability on very large (> 10k node) clusters.
The Serf changes:
- take snapshot disk IO out of the critical path for handling messages hashicorp/serf#524
- make snapshot compaction much less aggressive - the old fixed threshold caused snapshots to be constantly compacted (synchronously with request handling) on clusters larger than about 2000 nodes! hashicorp/serf#525
Memberlist changes:
- prioritize handling alive messages over suspect/dead to improve stability, and handle queue in LIFO order to avoid acting on info that 's already stale in the queue by the time we handle it. hashicorp/memberlist#159
- limit the number of concurrent pushPull requests being handled at once to 128. In one test scenario with 10s of thousands of servers we saw channel and lock blocking cause over 3000 pushPulls at once which ballooned the memory of the server because each push pull contained a de-serialised list of all known 10k+ nodes and their tags for a total of about 60 million objects and 7GB of memory stuck. While the rest of the fixes here should prevent the same root cause from blocking in the same way, this prevents any other bug or source of contention from allowing pushPull messages to stack up and eat resources. hashicorp/memberlist#158
* New Providers added and updated vendoring for go-discover
* Vendor.json formatted using make vendorfmt
* Docs/Agent/auto-join: Added documentation for the new providers introduced in this PR
* Updated the golang.org/x/sys/unix in the vendor directory
* Agent: TestGoDiscoverRegistration updated to reflect the addition of new providers
* Deleted terraform.tfstate from vendor.
* Deleted terraform.tfstate.backup
Deleted terraform state file artifacts from unknown runs.
* Updated x/sys/windows vendor for Windows binary compilation
The need has been spotted in issue https://github.com/hashicorp/consul/issues/3687.
Using "NYTimes/gziphandler", the http api responses can now be compressed if required.
The Go API requires compressed response if possible and handle the compressed response.
We here change only the http api (not the UI for instance).
Note that the vendor.json is already correct but the actual files were never checked in so report as missing:
```
$ govendor list | grep testify
v github.com/stretchr/testify/assert
m github.com/stretchr/testify/require
```
See https://github.com/hashicorp/consul/issues/3977
While trying to improve furthermore #3948 (This pull request is still valid since we are not using Compression to compute the result anyway).
I saw a strange behaviour of dns library.
Basically, msg.Len() and len(msg.Pack()) disagree on Message len.
Thus, calculation of DNS response is false consul relies on msg.Len() instead of the result of Pack()
This is linked to miekg/dns#453 and a fix has been provided with miekg/dns#454
Would it be possible to upgrade miekg/dns to a more recent function ?
Consul might for instance upgrade to a post 1.0 release such as https://github.com/miekg/dns/releases/tag/v1.0.4
Mitchell Hashimoto
Matt Keeler
Shubheksha
Paul Banks
Siva Prasad
Matt Keeler
Leo Zhang
Kyle Havlovitz
Jack Pearkes
Seth Vargo
Wim
Pierre Souchay
Preetha Appan
Yoann
Alvin Huang
James Phillips
Dmytro Kostiuchenko