Commit Graph

72 Commits

Author SHA1 Message Date
Pierre Souchay ae7f88f995 Avoid to have infinite recursion in DNS lookups when resolving CNAMEs (#4918)
* Avoid to have infinite recursion in DNS lookups when resolving CNAMEs

This will avoid killing Consul when a Service.Address is using CNAME
to a Consul CNAME that creates an infinite recursion.

This will fix https://github.com/hashicorp/consul/issues/4907

* Use maxRecursionLevel = 3 to allow several recursions
2019-01-07 16:53:54 -05:00
Paul Banks b29bc906ee
bugfix: use ServiceTags to generate cache key hash (#4987)
* bugfix: use ServiceTags to generate cahce key hash

* update unit test

* update

* remote print log

* Update .gitignore

* Completely deprecate ServiceTag field internally for clarity

* Add explicit test for CacheInfo cases
2019-01-07 21:30:47 +00:00
Pierre Souchay c5ae9caa28 Fixed another list of unstable unit tests in travis (#4915)
* Fixed another list of unstable unit tests in travis

Fixed failing tests in https://travis-ci.org/hashicorp/consul/jobs/451357061

* Fixed another list of unstable unit tests in travis.

Fixed failing tests in https://travis-ci.org/hashicorp/consul/jobs/451357061
2018-11-20 11:27:26 +00:00
Pierre Souchay fab55bee2b dns: implements prefix lookups for DNS TTL (#4605)
This will fix https://github.com/hashicorp/consul/issues/4509 and allow forinstance lb-* to match services lb-001 or lb-service-007.
2018-10-19 08:41:04 -07:00
Pierre Souchay 251156eb68 Added SOA configuration for DNS settings. (#4714)
This will allow to fine TUNE SOA settings sent by Consul in DNS responses,
for instance to be able to control negative ttl.

Will fix: https://github.com/hashicorp/consul/issues/4713

# Example

Override all settings:

* min_ttl: 0 => 60s
* retry: 600 (10m) => 300s (5 minutes),
* expire: 86400 (24h) => 43200 (12h)
* refresh: 3600 (1h) => 1800 (30 minutes)

```
consul agent -dev -hcl 'dns_config={soa={min_ttl=60,retry=300,expire=43200,refresh=1800}}'
```

Result:
```
dig +multiline @localhost -p 8600 service.consul

; <<>> DiG 9.12.1 <<>> +multiline @localhost -p 8600 service.consul
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 36557
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;service.consul.		IN A

;; AUTHORITY SECTION:
consul.			0 IN SOA ns.consul. hostmaster.consul. (
				1537959133 ; serial
				1800       ; refresh (30 minutes)
				300        ; retry (5 minutes)
				43200      ; expire (12 hours)
				60         ; minimum (1 minute)
				)

;; Query time: 4 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Wed Sep 26 12:52:13 CEST 2018
;; MSG SIZE  rcvd: 93
```
2018-10-10 15:50:56 -04:00
Paul Banks b83bbf248c Add Proxy Upstreams to Service Definition (#4639)
* Refactor Service Definition ProxyDestination.

This includes:
 - Refactoring all internal structs used
 - Updated tests for both deprecated and new input for:
   - Agent Services endpoint response
   - Agent Service endpoint response
   - Agent Register endpoint
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Register
     - Unmanaged deprecated field
     - Unmanaged new fields
     - Managed deprecated upstreams
     - Managed new
   - Catalog Services endpoint response
   - Catalog Node endpoint response
   - Catalog Service endpoint response
 - Updated API tests for all of the above too (both deprecated and new forms of register)

TODO:
 - config package changes for on-disk service definitions
 - proxy config endpoint
 - built-in proxy support for new fields

* Agent proxy config endpoint updated with upstreams

* Config file changes for upstreams.

* Add upstream opaque config and update all tests to ensure it works everywhere.

* Built in proxy working with new Upstreams config

* Command fixes and deprecations

* Fix key translation, upstream type defaults and a spate of other subtele bugs found with ned to end test scripts...

TODO: tests still failing on one case that needs a fix. I think it's key translation for upstreams nested in Managed proxy struct.

* Fix translated keys in API registration.
≈

* Fixes from docs
 - omit some empty undocumented fields in API
 - Bring back ServiceProxyDestination in Catalog responses to not break backwards compat - this was removed assuming it was only used internally.

* Documentation updates for Upstreams in service definition

* Fixes for tests broken by many refactors.

* Enable travis on f-connect branch in this branch too.

* Add consistent Deprecation comments to ProxyDestination uses

* Update version number on deprecation notices, and correct upstream datacenter field with explanation in docs
2018-10-10 16:55:34 +01:00
Matt Keeler d3ee66eed4
Add ECS option to EDNS responses where appropriate (#4647)
This implements parts of RFC 7871 where Consul is acting as an authoritative name server (or forwarding resolver when recursors are configured)

If ECS opt is present in the request we will mirror it back and return a response with a scope of 0 (global) or with the same prefix length as the request (indicating its valid specifically for that subnet).

We only mirror the prefix-length (non-global) for prepared queries as those could potentially use nearness checks that could be affected by the subnet. In the future we could get more sophisticated with determining the scope bits and allow for better caching of prepared queries that don’t rely on nearness checks.

The other thing this does not do is implement the part of the ECS RFC related to originating ECS headers when acting as a intermediate DNS server (forwarding resolver). That would take a quite a bit more effort and in general provide very little value. Consul will currently forward the ECS headers between recursors and the clients transparently, we just don't originate them for non-ECS clients to get potentially more accurate "location aware" results.
2018-09-11 09:37:46 -04:00
Pierre Souchay 22500f242e Fix unstable tests in agent, api, and command/watch 2018-09-10 16:58:53 +01:00
Pierre Souchay cec5d72396 BUGFIX: Unit test relying on WaitForLeader() did not work due to wrong test (#4472)
- Improve resilience of testrpc.WaitForLeader()

- Add additionall retry to CI

- Increase "go test" timeout to 8m

- Add wait for cluster leader to several tests in the agent package

- Add retry to some tests in the api and command packages
2018-08-06 19:46:09 -04:00
Siva Prasad 865068a358
DNS : Fixes recursors answering the DNS query to properly return the correct response. (#4461)
* Fixes the DNS recursor properly resolving the requests

* Added a test case for the recursor bug

* Refactored code && added a test case for all failing recursors

* Inner indentation moved into else if check
2018-08-02 10:12:52 -04:00
Matt Keeler 0fd7e97c2d Merge remote-tracking branch 'origin/master' into bugfix/prevent-multi-cname 2018-07-10 10:26:45 -04:00
Matt Keeler cbf8f14451 Ensure TXT RRs always end up in the Additional section except for ANY or TXT queries
This also changes where the enforcement of the enable_additional_node_meta_txt configuration gets applied.

formatNodeRecord returns the main RRs and the meta/TXT RRs in separate slices. Its then up to the caller to add to the appropriate sections or not.
2018-07-09 12:30:11 -04:00
Matt Keeler 22c2be5bf1 Fix some edge cases and add some tests. 2018-07-02 16:58:52 -04:00
mkeeler 6813a99081 Merge remote-tracking branch 'connect/f-connect' 2018-06-25 19:42:51 +00:00
Matt Keeler 8216816e3f Make filtering out TXT RRs only apply when they would end up in Additional section
ANY queries are no longer affected.
2018-06-19 10:08:16 -04:00
Mitchell Hashimoto 4d852e62a3
agent: address PR feedback 2018-06-14 09:41:49 -07:00
Mitchell Hashimoto 6313bc5615
agent: clarified a number of comments per PR feedback 2018-06-14 09:41:49 -07:00
Mitchell Hashimoto 7d79f9c46f
agent/consul: implement Health.ServiceNodes for Connect, DNS works 2018-06-14 09:41:47 -07:00
Mitchell Hashimoto 406366c45b
agent: working DNS for Connect queries, I think, but have to
implement Health endpoints to be sure
2018-06-14 09:41:47 -07:00
Matt Keeler 6cc0422408 Add configuration entry to control including TXT records for node meta in DNS responses
If set to false, the only way to retrieve TXT records for node meta is to specifically query for TXT records.
2018-06-11 11:49:04 -04:00
Matt Keeler 27fe219918
Merge pull request #4131 from pierresouchay/enable_full_dns_compression
Enable full dns compression
2018-06-01 10:42:03 -04:00
Wim 16ce8d9ed2 Add service reverse lookup tests 2018-05-21 22:59:21 +02:00
Pierre Souchay bb92420873 Test fix, trying to pass Travis tests 2018-05-16 14:10:35 +02:00
Pierre Souchay cfa5986df7 Fixed unit tests and updated limits 2018-05-16 12:11:49 +02:00
Pierre Souchay c715408c87 More Tests cases compression/no compression 2018-04-21 17:18:39 +02:00
Pierre Souchay 36827418b7 Improved unit tests debug info when it fails 2018-04-18 14:18:17 +02:00
Pierre Souchay 9243daeb0e Run new test in parallel 2018-04-17 10:36:12 +02:00
Pierre Souchay 5b4905e11d More test cases + travis flacky 2018-04-17 09:42:08 +02:00
Pierre Souchay fadfb95e07 Added Unit tests + fixed boudary limit 2018-04-17 09:31:30 +02:00
Kyle Havlovitz af4be34a2a
Update make static-assets goal and run format 2018-04-13 09:57:25 -07:00
Matt Keeler 0619efc254 GH-3798: More PR Updates
Update docs a little
Update/add tests. Make sure all the various ways of determining the source IP work
Update X-Forwarded-For header parsing. This can be a comma separated list with the first element being the original IP so we now handle csv data there.
Got rid of error return from sourceAddrFromRequest
2018-04-12 10:40:46 -04:00
Matt Keeler 136efeb3be GH-3798: A couple more PR updates
Test HTTP/DNS source IP without header/extra EDNS data.
Add WARN log for when prepared query with near=_ip is executed without specifying the source ip
2018-04-12 10:10:37 -04:00
Matt Keeler cec8d5145b GH-3798: A few more PR updates 2018-04-11 20:32:35 -04:00
Matt Keeler d065d3a6db GH-3798: Updates for PR
Allow DNS peer IP as the source IP.
Break early when the right node was found for executing the preapred query.
Update docs
2018-04-11 17:02:04 -04:00
Matt Keeler 283a7942c4 GH-3798: Wrap DNS request validation in a retry 2018-04-11 16:00:15 -04:00
Matt Keeler 5794fa8837 GH-3798: Add DNS near=_ip test 2018-04-11 10:33:48 -04:00
Matt Keeler 27899f9f46
Merge pull request #3948 from pierresouchay/fix_tcp_dns_limit
[BUGFIX] do not break when TCP DNS answer exceeds 64k
2018-03-30 16:25:23 -04:00
Matt Keeler ebc6f414ac Formatting update 2018-03-27 16:31:27 -04:00
Josh Soref 94835a2715 Spelling (#3958)
* spelling: another

* spelling: autopilot

* spelling: beginning

* spelling: circonus

* spelling: default

* spelling: definition

* spelling: distance

* spelling: encountered

* spelling: enterprise

* spelling: expands

* spelling: exits

* spelling: formatting

* spelling: health

* spelling: hierarchy

* spelling: imposed

* spelling: independence

* spelling: inspect

* spelling: last

* spelling: latest

* spelling: client

* spelling: message

* spelling: minimum

* spelling: notify

* spelling: nonexistent

* spelling: operator

* spelling: payload

* spelling: preceded

* spelling: prepared

* spelling: programmatically

* spelling: required

* spelling: reconcile

* spelling: responses

* spelling: request

* spelling: response

* spelling: results

* spelling: retrieve

* spelling: service

* spelling: significantly

* spelling: specifies

* spelling: supported

* spelling: synchronization

* spelling: synchronous

* spelling: themselves

* spelling: unexpected

* spelling: validations

* spelling: value
2018-03-19 16:56:00 +00:00
Pierre Souchay aebfcb6767 Fixed minor typo (+ travis tests is unstable) 2018-03-09 18:42:13 +01:00
Pierre Souchay 93fa1f6f49 Optimize size for SRV records, should improve performance a bit
Stricter Unit tests that checks if truncation was OK.
2018-03-09 18:25:29 +01:00
Pierre Souchay d0e45f22df Fixed wrong format of debug msg in unit test 2018-03-08 00:36:17 +01:00
Pierre Souchay 7d59249d96 Avoid issue with compression of DNS messages causing overflow 2018-03-07 23:33:41 +01:00
Pierre Souchay 419bf29041 Cleaner Unit tests from suggestions from @preetapan 2018-03-07 18:24:41 +01:00
Pierre Souchay be39fb20cc [BUGFIX] do not break when TCP DNS answer exceeds 64k
It will avoid having discovery broken when having large number
of instances of a service (works with SRV and A* records).

Fixes https://github.com/hashicorp/consul/issues/3850
2018-03-07 10:08:06 +01:00
Pierre Souchay 0b7f620dc6 Allow to control the number of A/AAAA Record returned by DNS
This allows to have randomized resource records (i.e. each
answer contains only one IP, but the IP changes every request) for
A, AAAA records.

It will fix https://github.com/hashicorp/consul/issues/3355 and
https://github.com/hashicorp/consul/issues/3937

See https://github.com/hashicorp/consul/issues/3937#issuecomment-370610509
for details.

It basically add a new option called `a_record_limit` and will not
return more than a_record_limit when performing A, AAAA or ANY DNS
requests.

The existing `udp_answer_limit` option is still working but should
be considered as deprecated since it works only with DNS clients
not supporting EDNS.
2018-03-06 02:07:42 +01:00
James Phillips 3724e49ddf
Fixes a panic on TCP-based DNS lookups.
This came in via the monkey patch in #3861.

Fixes #3877
2018-02-08 17:57:41 -08:00
Veselkov Konstantin 7de57ba4de remove golint warnings 2018-01-28 22:40:13 +04:00
Frank Schröder 350932161d dns: return NXDOMAIN if datacenter is invalid (#3200) (#3596)
Queries to the DNS server can contain an optional datacenter
name in the query name. You can query for 'foo.service.consul'
or 'foo.service.dc.consul' to get a response for either the
default or a specific datacenter.

Datacenter names cannot have dots, therefore the datacenter
name can refer to only one element in the DNS query name.

The DNS server allowed extra labels between the optional
datacenter name and the domain and returned a valid response
instead of returning NXDOMAIN. For example, if the domain
is set to '.consul' then 'foo.service.dc1.extra.consul'
should return NXDOMAIN because of 'extra' being between
the datacenter name 'dc1' and the domain '.consul'.

Fixes #3200
2017-10-20 16:49:17 -07:00
Ryan Slade 85e4aea9d1 Replace time.Now().Sub(x) with time.Since(x) 2017-10-17 20:38:24 +02:00