Commit Graph

726 Commits

Author SHA1 Message Date
Pierre Souchay c152cb7bdf Added Missing Service Meta synchronization and field 2018-04-21 17:34:29 +02:00
Pierre Souchay c715408c87 More Tests cases compression/no compression 2018-04-21 17:18:39 +02:00
Pierre Souchay 076ecf9712 Removed unecessary copy of Extra and index 2018-04-20 22:51:04 +02:00
Pierre Souchay 06a181955d Use safer stringVal() 2018-04-18 23:18:16 +02:00
Pierre Souchay 9bb15730a6 Added unit test on key length 2018-04-18 23:07:25 +02:00
Pierre Souchay 2f5e67534d Added unit tests for bad meta values 2018-04-18 22:57:33 +02:00
Pierre Souchay d2ab3deacf [BUGFIX] Added Service Meta support in configuration files
Fixes https://github.com/hashicorp/consul/issues/4045

Was not added by mistake in https://github.com/hashicorp/consul/pull/3881
2018-04-18 22:18:58 +02:00
Pierre Souchay 36827418b7 Improved unit tests debug info when it fails 2018-04-18 14:18:17 +02:00
Pierre Souchay 728c5308df Fixed sync of Extra in binarySearch 2018-04-18 14:17:44 +02:00
Pierre Souchay 9243daeb0e Run new test in parallel 2018-04-17 10:36:12 +02:00
Pierre Souchay 5b4905e11d More test cases + travis flacky 2018-04-17 09:42:08 +02:00
Pierre Souchay fadfb95e07 Added Unit tests + fixed boudary limit 2018-04-17 09:31:30 +02:00
Pierre Souchay c838376dfa Added comment for function dnsBinaryTruncate 2018-04-17 01:10:52 +02:00
Pierre Souchay 94c0bf978a Perform a binary search to find optimal size of DNS responses
Will fix https://github.com/hashicorp/consul/issues/4036

Instead of removing one by one the entries, find the optimal
size using binary search.

For SRV records, with 5k nodes, duration of DNS lookups is
divided by 4 or more.
2018-04-17 00:50:00 +02:00
Kyle Havlovitz 2a636275ad
Update static assets 2018-04-13 10:05:30 -07:00
Kyle Havlovitz af4be34a2a
Update make static-assets goal and run format 2018-04-13 09:57:25 -07:00
Matt Keeler d926679278
Merge pull request #4023 from hashicorp/f-near-ip
Add near=_ip support for prepared queries
2018-04-12 12:10:48 -04:00
Matt Keeler 0619efc254 GH-3798: More PR Updates
Update docs a little
Update/add tests. Make sure all the various ways of determining the source IP work
Update X-Forwarded-For header parsing. This can be a comma separated list with the first element being the original IP so we now handle csv data there.
Got rid of error return from sourceAddrFromRequest
2018-04-12 10:40:46 -04:00
Matt Keeler 136efeb3be GH-3798: A couple more PR updates
Test HTTP/DNS source IP without header/extra EDNS data.
Add WARN log for when prepared query with near=_ip is executed without specifying the source ip
2018-04-12 10:10:37 -04:00
Matt Keeler cec8d5145b GH-3798: A few more PR updates 2018-04-11 20:32:35 -04:00
Matt Keeler d065d3a6db GH-3798: Updates for PR
Allow DNS peer IP as the source IP.
Break early when the right node was found for executing the preapred query.
Update docs
2018-04-11 17:02:04 -04:00
Matt Keeler 283a7942c4 GH-3798: Wrap DNS request validation in a retry 2018-04-11 16:00:15 -04:00
Jack Pearkes 265359959b
Merge pull request #4015 from hashicorp/ui-service-tags
api/ui: return tags on internal UI endpoints
2018-04-11 12:02:19 -07:00
Matt Keeler 5794fa8837 GH-3798: Add DNS near=_ip test 2018-04-11 10:33:48 -04:00
Matt Keeler de403d6515 GH-3798: Add HTTP prepared query near=_ip test
Also fixed an issue where we need to have the X-Forwarded-For header processed before the RemoteAddr. This shouldn’t have any functional difference for prod code but for mocked request objects it allows them to work.
2018-04-10 15:35:54 -04:00
Matt Keeler 45a537def9 GH-3798: Add near=_ip support for prepared queries 2018-04-10 14:50:50 -04:00
Isaac Williams 01f5db46e8 Close HTTP response in Agent test (HTTPAPI_MethodNotAllowed_OSS) 2018-04-10 13:18:46 -04:00
Paul Banks 0d8993e338
Allow ignoring checks by ID when defining a PreparedQuery. Fixes #3727. 2018-04-10 14:04:16 +01:00
Pierre Souchay a680c8e91b Clearer documentation and comments for enabling Prometheus support 2018-04-09 13:16:45 +02:00
Pierre Souchay 27362320e8 Enable compression / automatic Mime-Type detection for Prometheus endpoint 2018-04-09 13:16:03 +02:00
Matt Keeler 39c17084b6
Merge pull request #4006 from kjothen/patch-1
Update check.go
2018-04-06 12:57:52 -04:00
Pierre Souchay 93a01b0949 Now use prometheus_retention_time > 0 to enable prometheus support 2018-04-06 14:21:05 +02:00
Pierre Souchay fd98fb1449 Added support exposing metrics in Prometheus format 2018-04-06 09:18:06 +02:00
Jack Pearkes eb447f51e6 api/ui: return tags on internal UI endpoints
This is to allow the UI to display tags in the services index pages
without needing to make additional queries.
2018-04-05 12:28:57 -07:00
Matt Keeler 0d1d03c793
Merge pull request #3752 from yfouquet/issue_3687
Add support for compression in http api
2018-04-04 09:06:42 -04:00
Yoann 0f6e05d4c1 Add support for compression in http api
The need has been spotted in issue https://github.com/hashicorp/consul/issues/3687.
Using "NYTimes/gziphandler", the http api responses can now be compressed if required.
The Go API requires compressed response if possible and handle the compressed response.
We here change only the http api (not the UI for instance).
2018-04-03 22:33:13 +02:00
Preetha 2c931c92d4
Merge pull request #3998 from zte-opensource/wip-fix-shutdown
minor fix for endpoints shutdown
2018-04-03 12:22:54 -05:00
Kieran Othen e4b7465193
Update check.go
Cosmetic fix to the agent's HTTP check function which always formats the result as "HTTP GET ...", ignoring any non-GET supplied HTTP method such as POST, PUT, etc.
2018-03-31 16:44:35 +01:00
Matt Keeler 27899f9f46
Merge pull request #3948 from pierresouchay/fix_tcp_dns_limit
[BUGFIX] do not break when TCP DNS answer exceeds 64k
2018-03-30 16:25:23 -04:00
Preetha a67d27c756
Adds discovery_max_stale (#4004)
Adds a new option to allow service discovery endpoints to return stale results if configured at the agent level.
2018-03-30 10:14:44 -05:00
Preetha 1609b2c6b7
Merge pull request #3994 from hashicorp/f-rename-servicemeta
Renames agent API layer for service metadata to "meta" for consistency
2018-03-29 14:07:57 -05:00
runsisi b082c8d3ab minor fix for endpoints shutdown
Signed-off-by: runsisi <runsisi@zte.com.cn>
2018-03-29 21:45:46 +08:00
Matt Keeler ea6767d8bc
Merge pull request #3990 from hashicorp/b-gh-3854
Warn when node name isnt a valid DNS label
2018-03-29 09:04:47 -04:00
Preetha Appan c7581d68c6
Renames agent API layer for service metadata to "meta" for consistency 2018-03-28 09:04:50 -05:00
Preetha daa61c5803
Merge pull request #3881 from pierresouchay/service_metadata
Feature Request: Support key-value attributes for services
2018-03-27 16:33:57 -05:00
Preetha 3c96d64eaa
Merge pull request #3984 from hashicorp/f-allow-federation-disable
Allows disabling WAN federation by setting serf WAN port to -1
2018-03-27 16:05:53 -05:00
Matt Keeler ebc6f414ac Formatting update 2018-03-27 16:31:27 -04:00
Pierre Souchay 980189a33f Added validation of ServiceMeta in Catalog
Fixed Error Message when ServiceMeta is not valid

Added Unit test for adding a Service with badly formatted ServiceMeta
2018-03-27 22:22:42 +02:00
Preetha Appan 226cb2e95c
fix typo and remove comment 2018-03-27 14:28:05 -05:00
Matt Keeler 3facce6bfa GH-3854: Warn when node name isnt a valid DNS label 2018-03-27 15:00:33 -04:00
Preetha Appan 010a459365
Remove unnecessary nil checks 2018-03-27 10:59:42 -05:00
Preetha Appan 6c0bb5a810
Fix test and remove unused method 2018-03-27 09:44:41 -05:00
Preetha Appan d77ab91123
Allows disabling WAN federation by setting serf WAN port to -1 2018-03-26 14:21:06 -05:00
Pierre Souchay 18baff80ae Merge remote-tracking branch 'origin/master' into node_health_should_change_service_index 2018-03-22 13:07:11 +01:00
Pierre Souchay 5fb1b18073 More test cases 2018-03-22 12:41:06 +01:00
Pierre Souchay 39a7b5c20d Added new test regarding checks index 2018-03-22 12:20:25 +01:00
Pierre Souchay dd9efb755a Fixed minor typo in comments
Might fix unstable travis build
2018-03-22 10:30:10 +01:00
Guido Iaquinti 8cd11d5888 Add package name to log output 2018-03-21 15:56:14 +00:00
Josh Soref 94835a2715 Spelling (#3958)
* spelling: another

* spelling: autopilot

* spelling: beginning

* spelling: circonus

* spelling: default

* spelling: definition

* spelling: distance

* spelling: encountered

* spelling: enterprise

* spelling: expands

* spelling: exits

* spelling: formatting

* spelling: health

* spelling: hierarchy

* spelling: imposed

* spelling: independence

* spelling: inspect

* spelling: last

* spelling: latest

* spelling: client

* spelling: message

* spelling: minimum

* spelling: notify

* spelling: nonexistent

* spelling: operator

* spelling: payload

* spelling: preceded

* spelling: prepared

* spelling: programmatically

* spelling: required

* spelling: reconcile

* spelling: responses

* spelling: request

* spelling: response

* spelling: results

* spelling: retrieve

* spelling: service

* spelling: significantly

* spelling: specifies

* spelling: supported

* spelling: synchronization

* spelling: synchronous

* spelling: themselves

* spelling: unexpected

* spelling: validations

* spelling: value
2018-03-19 16:56:00 +00:00
Paul Banks a8f7681c70
Merge pull request #3962 from canterberry/upgrade/tls-cipher-suites
🔒 Update supported TLS cipher suites
2018-03-19 16:44:33 +00:00
Pierre Souchay b6914617d9 Fixed typo in comments 2018-03-19 17:12:08 +01:00
Pierre Souchay 5e974843f1 Refactoring to have clearer code without weird bool 2018-03-19 16:12:54 +01:00
Pierre Souchay a44b9e84b1 [BUGFIX] When a node level check is removed, ensure all services of node are notified
Bugfix for https://github.com/hashicorp/consul/pull/3899

When a node level check is removed (example: maintenance),
some watchers on services might have to recompute their state.

If those nodes are performing blocking queries, they have to be notified.
While their state was updated when node-level state did change or was added
this was not the case when the check was removed. This fixes it.
2018-03-19 14:14:03 +01:00
Preetha Appan 2eed7766a8
cleanup unit test code a bit 2018-03-16 09:36:57 -05:00
Preetha c87699abf2
Merge pull request #3885 from eddsteel/support-options-requests
Support OPTIONS requests
2018-03-16 09:20:16 -05:00
Devin Canterberry 2187ab1e1c
🎨 Formatting changes only; convert leading space to tabs 2018-03-15 10:30:38 -07:00
Devin Canterberry 7236c95e11
Match expectation of TLSCipherSuites to values of tls_cipher_suites 2018-03-15 10:19:46 -07:00
Devin Canterberry a61abcd931
🐛 Formatting changes only; add missing trailing commas 2018-03-15 10:19:46 -07:00
Devin Canterberry c901307a47
🔒 Update supported TLS cipher suites
The list of cipher suites included in this commit are consistent with
the values and precedence in the [Golang TLS documentation](https://golang.org/src/crypto/tls/cipher_suites.go).

> **Note:** Cipher suites with RC4 are still included within the list
> of accepted values for compatibility, but **these cipher suites are
> not safe to use** and should be deprecated with warnings and
> subsequently removed. Support for RC4 ciphers has already been
> removed or disabled by default in many prominent browsers and tools,
> including Golang.
>
> **References:**
>
>  * [RC4 on Wikipedia](https://en.wikipedia.org/wiki/RC4)
>  * [Mozilla Security Blog](https://blog.mozilla.org/security/2015/09/11/deprecating-the-rc4-cipher/)
2018-03-15 10:19:46 -07:00
Pierre Souchay aebfcb6767 Fixed minor typo (+ travis tests is unstable) 2018-03-09 18:42:13 +01:00
Pierre Souchay 93fa1f6f49 Optimize size for SRV records, should improve performance a bit
Stricter Unit tests that checks if truncation was OK.
2018-03-09 18:25:29 +01:00
Preetha 210cfe5ef9
Merge pull request #3940 from pierresouchay/dns_max_size
Allow to control the number of A/AAAA Record returned by DNS
2018-03-09 07:35:32 -06:00
Pierre Souchay d0e45f22df Fixed wrong format of debug msg in unit test 2018-03-08 00:36:17 +01:00
Pierre Souchay ce3f47a75d Performance optimization for services having more than 2k records 2018-03-08 00:26:41 +01:00
Pierre Souchay 7d59249d96 Avoid issue with compression of DNS messages causing overflow 2018-03-07 23:33:41 +01:00
Pierre Souchay 419bf29041 Cleaner Unit tests from suggestions from @preetapan 2018-03-07 18:24:41 +01:00
Pierre Souchay b77fd5ce9d 64000 max limit to DNS messages since there is overhead
Added debug log to give information about truncation.
2018-03-07 16:14:41 +01:00
Pierre Souchay be39fb20cc [BUGFIX] do not break when TCP DNS answer exceeds 64k
It will avoid having discovery broken when having large number
of instances of a service (works with SRV and A* records).

Fixes https://github.com/hashicorp/consul/issues/3850
2018-03-07 10:08:06 +01:00
Mitchell Hashimoto 8217564c48
agent/consul/fsm: begin using testify/assert 2018-03-06 09:48:15 -08:00
Pierre Souchay 0b7f620dc6 Allow to control the number of A/AAAA Record returned by DNS
This allows to have randomized resource records (i.e. each
answer contains only one IP, but the IP changes every request) for
A, AAAA records.

It will fix https://github.com/hashicorp/consul/issues/3355 and
https://github.com/hashicorp/consul/issues/3937

See https://github.com/hashicorp/consul/issues/3937#issuecomment-370610509
for details.

It basically add a new option called `a_record_limit` and will not
return more than a_record_limit when performing A, AAAA or ANY DNS
requests.

The existing `udp_answer_limit` option is still working but should
be considered as deprecated since it works only with DNS clients
not supporting EDNS.
2018-03-06 02:07:42 +01:00
Edd Steel 41b1d45cc7
Re-use defined endpoints for tests 2018-03-03 11:19:18 -08:00
Paul Banks 9a47449c6d
Merge pull request #3899 from pierresouchay/fix_blocking_queries_index
Services Indexes modified per service instead of using a global Index
2018-03-02 16:24:43 +00:00
Pierre Souchay 360dc1dd8d Simplified error handling for maxIndexForService
* added unit tests to ensure service index is properly garbage collected
* added Upgrade from Version 1.0.6 to higher section in documentation
2018-03-01 14:09:36 +01:00
Paul Banks dbaabb1dbc
Fix test running in non-bash shells 2018-02-22 14:06:06 +00:00
Paul Banks 6da6e086ef
Merge pull request #3900 from hashicorp/fix-monitor-sigint-3891
Fixes #3891: agent monitor no longer unresponsive before logs stream.
2018-02-21 21:28:33 +00:00
Preetha Appan 80791d5b21
Remove extra newline 2018-02-21 13:21:47 -06:00
Preetha Appan 907b97b7f2
Unit test that calls revokeLeadership twice to make sure its idempotent 2018-02-21 12:48:53 -06:00
Preetha Appan f59abcc394
Make sure revokeLeadership is called if establishLeadership errors 2018-02-21 12:33:22 -06:00
Alex Dadgar 18bf9647d5 Test autopilots start/stop idempotency 2018-02-21 10:19:30 -08:00
Alex Dadgar 33c5afdb31 Improve autopilot shutdown to be idempotent 2018-02-20 15:51:59 -08:00
Pierre Souchay a8d3745104 Fixed comments for function maxIndexForService 2018-02-20 23:57:28 +01:00
Pierre Souchay 09351ba9a6 [Revert] Only update services if tags are different
This patch did give some better results, but break watches on
the services of a node.

It is possible to apply the same optimization for nodes than
to services (one index per instance), but it would complicate
further the patch.

Let's do it in another PR.
2018-02-20 23:34:42 +01:00
Pierre Souchay 60454b570a Only update services if tags are different 2018-02-20 23:08:04 +01:00
Pierre Souchay a05d38737c Enable Raft index optimization per service name on health endpoint
Had to fix unit test in order to check properly indexes.
2018-02-20 01:35:50 +01:00
Paul Banks de58eb1820
Fixes #3891: agent monitor no longer unresponsive before logs stream.
The root cause is actually that the agent's streaming HTTP API didn't flush until the first log line was found which commonly was pretty soon since the default level is INFO. In cases where there were no logs immediately due to level for instance, the client gets stuck in the HTTP code waiting on a response packet from the server before we enter the loop that checks the shutdown channel from the signal handler.

This fix flushes the initial status immediately on the streaming endpoint which lets the client code get into it's expected state where it's listening for shutdown or log lines.
2018-02-19 21:53:10 +00:00
Pierre Souchay 4f10fae3c3 Get only first service to test whether we have to cleanup index of a service 2018-02-19 22:44:49 +01:00
Pierre Souchay bac8fb046f Fixed comment about raftIndex + use test.Helper() 2018-02-19 19:30:25 +01:00
Pierre Souchay 73127ef407 Services Indexes modified per service instead of using a global Index
This patch improves the watches for services on large cluster:
each service has now its own index, such watches on a specific service
are not modified by changes in the global catalog.

It should improve a lot the performance of tools such as consul-template
or libraries performing watches on very large clusters with many
services/watches.
2018-02-19 18:29:22 +01:00
Edd Steel d0f0d67b4a
Clarify comments 2018-02-17 17:46:11 -08:00
Edd Steel f770f360e9 Test every endpoint for OPTIONS/MethodNotFound 2018-02-17 17:34:13 -08:00