Commit Graph

2190 Commits

Author SHA1 Message Date
Hans Hasselberg ae2cbbce99 agent/cache test for cache throttling. (#8396) 2020-07-30 12:41:38 +00:00
Matt Keeler e813445e57 Agent Auto Config: Implement Certificate Generation (#8360)
Most of the groundwork was laid in previous PRs between adding the cert-monitor package to extracting the logic of signing certificates out of the connect_ca_endpoint.go code and into a method on the server.

This also refactors the auto-config package a bit to split things out into multiple files.
2020-07-28 19:32:22 +00:00
Matt Keeler 91ec880e07
Backport #8389 (#8392)
# Conflicts:
#	agent/cache-types/catalog_list_services_test.go
2020-07-28 14:22:29 -04:00
Pierre Souchay 678489d9d1 Added ratelimit to handle throtling cache (#8226)
This implements a solution for #7863

It does:

    Add a new config cache.entry_fetch_rate to limit the number of calls/s for a given cache entry, default value = rate.Inf
    Add cache.entry_fetch_max_burst size of rate limit (default value = 2)

The new configuration now supports the following syntax for instance to allow 1 query every 3s:

    command line HCL: -hcl 'cache = { entry_fetch_rate = 0.333}'
    in JSON

{
  "cache": {
    "entry_fetch_rate": 0.333
  }
}
2020-07-27 21:11:42 +00:00
Matt Keeler 0937f70ddf Move connect root retrieval and cert signing logic out of the RPC endpoints (#8364)
The code now lives on the Server type itself. This was done so that all of this could be shared with auto config certificate signing.
2020-07-24 14:01:58 +00:00
Matt Keeler 4d41ee3887 Move generation of the CA Configuration from the agent code into a method on the RuntimeConfig (#8363)
This allows this to be reused elsewhere.
2020-07-23 20:05:52 +00:00
Matt Keeler 56b46436c1
Backport: #8362 (#8366)
Refactoring of the agentpb package.

First move the whole thing to the top-level proto package name.

Secondly change some things around internally to have sub-packages.
# Conflicts:
#	agent/consul/state/acl.go
#	agent/consul/state/acl_test.go
2020-07-23 12:44:27 -04:00
Daniel Nephin 4205fdf1d6 Merge pull request #7948 from hashicorp/dnephin/buffer-test-logs
testutil: NewLogBuffer - buffer logs until a test fails
2020-07-21 19:22:29 +00:00
Matt Keeler a8d2e5a2c2
Disable background cache refresh for Connect Leaf Certs
The rationale behind removing them is that all of our own code (xDS, builtin connect proxy) use the cache notification mechanism. This ensures that the blocking fetch behind the scenes is always executing. Therefore the only way you might go to get a certificate and have to wait is when 1) the request has never been made for that cert before or 2) you are using the v1/agent/connect/ca/leaf API for retrieving the cert yourself.

In the first case, the refresh change doesn’t alter the behavior. In the second case, it can be mitigated by using blocking queries with that API which just like normal cache notification mechanism will cause the blocking fetch to be initiated and to get leaf certs as soon as needed.

If you are not using blocking queries, or Envoy/xDS, or the builtin connect proxy but are retrieving the certs yourself then the HTTP endpoint might take a little longer to respond.

This also renames the RefreshTimeout field on the register options to QueryTimeout to more accurately reflect that it is used for any type that supports blocking queries.

# Conflicts:
#	agent/cache/cache.go
2020-07-21 13:51:18 -04:00
Matt Keeler 24e11b511e
Fix issue with changing the agent token causing failure to renew the auto-encrypt certificate
The fallback method would still work but it would get into a state where it would let the certificate expire for 10s before getting a new one. And the new one used the less secure RPC endpoint.

This is also a pretty large refactoring of the auto encrypt code. I was going to write some tests around the certificate monitoring but it was going to be impossible to get a TestAgent configured in such a way that I could write a test that ran in less than an hour or two to exercise the functionality.

Moving the certificate monitoring into its own package will allow for dependency injection and in particular mocking the cache types to control how it hands back certificates and how long those certificates should live. This will allow for exercising the main loop more than would be possible with it coupled so tightly with the Agent.

# Conflicts:
#	agent/agent.go
2020-07-21 13:49:18 -04:00
Daniel Nephin 65566e2c98 Merge pull request #8290 from hashicorp/dnephin/watch-decode
watch: fix script watches with single arg
2020-07-20 18:41:48 +00:00
André 927e73d8db minor: fix docstring of DNSOnlyPassing (#8318)
In runtime.go it had "duration" but it is actually a boolean.
2020-07-16 13:48:07 +00:00
Matt Keeler 625055a556 Add ability for notifications when one of the agent tokens is updated (#8301)
Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
2020-07-14 13:54:38 +00:00
Freddy 89af0212d3 Add api mod support for /catalog/gateway-services (#8278) 2020-07-10 19:02:09 +00:00
R.B. Boyer 2142a697ad
[backport: 1.8.x] xds: version sniff envoy and switch regular expressions from 'regex' to 'safe_regex' on newer envoy versions (#8265)
cherry-pick of #8222 onto origin/release/1.8.x

Fixes: #8205
2020-07-09 17:04:23 -05:00
Matt Keeler 38251ab0e8
Pass the Config and TLS Configurator into the AutoConfig constructor
This is instead of having the AutoConfigBackend interface provide functions for retrieving them.

NOTE: the config is not reloadable. For now this is fine as we don’t look at any reloadable fields. If that changes then we should provide a way to make it reloadable.
2020-07-09 10:38:29 -04:00
Matt Keeler f06595992a
Rename (*Server).forward to (*Server).ForwardRPC
Also get rid of the preexisting shim in server.go that existed before to have this name just call the unexported one.
2020-07-09 10:38:16 -04:00
Matt Keeler 977eb725a7
Refactor AutoConfig RPC to not have a direct dependency on the Server type
Instead it has an interface which can be mocked for better unit testing that is deterministic and not prone to flakiness.

# Conflicts:
#	agent/pool/pool.go
2020-07-09 10:37:55 -04:00
R.B. Boyer 8a5680aaf0
connect: upgrade github.com/envoyproxy/go-control-plane to v0.9.5 (#8247)
cherry-pick of #8165 onto origin/release/1.8.x
2020-07-07 16:22:30 -05:00
Chris Piraino cbf143844f Append port number to ingress host domain (#8190)
A port can be sent in the Host header as defined in the HTTP RFC, so we
take any hosts that we want to match traffic to and also add another
host with the listener port added.

Also fix an issue with envoy integration tests not running the
case-ingress-gateway-tls test.
2020-07-07 15:43:32 +00:00
Matt Keeler 9c64239db7 Merge pull request #8211 from hashicorp/bugfix/auto-encrypt-various 2020-07-02 13:51:34 +00:00
Matt Keeler d73d299848 Merge pull request #8218 from yurkeen/fix-dns-rcode 2020-07-01 13:13:55 +00:00
Matt Keeler c7a6c5c4f5 Merge pull request #8193 from hashicorp/feature/auto-config/suppress-config-warnings 2020-06-27 14:07:30 +00:00
Freddy be263d7885 Split up unused key validation for oss/ent (#8189)
Split up unused key validation in config entry decode for oss/ent.

This is needed so that we can return an informative error in OSS if namespaces are provided.
2020-06-26 12:02:56 +02:00
Matt Keeler 8853e38c72
Various go routine leak fixes 2020-06-25 09:36:14 -04:00
Chris Piraino 3da13af6b4 Merge pull request #7932 from hashicorp/ingress/internal-ui-endpoint-multiple-ports
Update gateway-services-nodes API endpoint to allow multiple addresses
2020-06-24 22:11:45 +00:00
Matt Keeler 1858153500 Don’t leak metrics go routines in tests (#8182) 2020-06-24 14:15:50 +00:00
gitforbit 657db029b2 agent-http: cleanup: return nil instead of err (#8043)
Since err is already checked, it should return `nil`
2020-06-24 12:29:48 +00:00
Freddy fc1baf2223 Merge pull request #8169 from hashicorp/config-entry-ns 2020-06-23 11:44:57 -06:00
Pierre Souchay 9df55f5995 Returns DNS Error NSDOMAIN when DC does not exists (#8103)
This will allow to increase cache value when DC is not valid (aka
return SOA to avoid too many consecutive requests) and will
distinguish DC being temporarily not available from DC not existing.

Implements https://github.com/hashicorp/consul/issues/8102
2020-06-22 13:02:47 +00:00
Matt Keeler 3f2fc48623 Require enabling TLS to enable Auto Config (#8159)
On the servers they must have a certificate.

On the clients they just have to set verify_outgoing to true to attempt TLS connections for RPCs.

Eventually we may relax these restrictions but right now all of the settings we push down (acl tokens, acl related settings, certificates, gossip key) are sensitive and shouldn’t be transmitted over an unencrypted connection. Our guides and docs should recoommend verify_server_hostname on the clients as well.

Another reason to do this is weird things happen when making an insecure RPC when TLS is not enabled. Basically it tries TLS anyways. We should probably fix that to make it clearer what is going on.
2020-06-19 20:38:38 +00:00
Freddy dce775d0d8 Always return a gateway cluster (#8158) 2020-06-19 19:32:24 +00:00
Matt Keeler 0736c42b72 Allow cancelling startup when performing auto-config (#8157)
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2020-06-19 19:16:20 +00:00
Matt Keeler fdef446e82 Change auto config authorizer to allow for future extension
The envisioned changes would allow extra settings to enable dynamically defined auth methods to be used instead of  or in addition to the statically defined one in the configuration.
2020-06-18 19:22:51 +00:00
Chris Piraino 8d72225d33 Remove ACLEnforceVersion8 from tests (#8138)
The field had been deprecated for a while and was recently removed,
however a PR which added these tests prior to removal was merged.
2020-06-18 18:15:43 +00:00
Matt Keeler 6375db7b4b Merge pull request #8086 from hashicorp/feature/auto-config/client-config-inject 2020-06-18 14:45:52 +00:00
Matt Keeler 9f37a218c5 Merge pull request #8035 from hashicorp/feature/auto-config/server-rpc 2020-06-17 20:08:17 +00:00
Daniel Nephin 058114e82e Merge pull request #7762 from hashicorp/dnephin/warn-on-unknown-service-file
config: warn if a config file is being skipped because of its file extension
2020-06-17 15:21:34 -04:00
Pierre Souchay 318495d1f8 gossip: Ensure that metadata of Consul Service is updated (#7903)
While upgrading servers to a new version, I saw that metadata of
existing servers are not upgraded, so the version and raft meta
is not up to date in catalog.

The only way to do it was to:
 * update Consul server
 * make it leave the cluster, then metadata is accurate

That's because the optimization to avoid updating catalog does
not take into account metadata, so no update on catalog is performed.
2020-06-17 10:17:33 +00:00
Matt Keeler c3b348bebb Agent Auto Configuration: Configuration Syntax Updates (#8003) 2020-06-16 19:03:59 +00:00
Matt Keeler 3c4413cbed ACL Node Identities (#7970)
A Node Identity is very similar to a service identity. Its main targeted use is to allow creating tokens for use by Consul agents that will grant the necessary permissions for all the typical agent operations (node registration, coordinate updates, anti-entropy).

Half of this commit is for golden file based tests of the acl token and role cli output. Another big updates was to refactor many of the tests in agent/consul/acl_endpoint_test.go to use the same style of tests and the same helpers. Besides being less boiler plate in the tests it also uses a common way of starting a test server with ACLs that should operate without any warnings regarding deprecated non-uuid master tokens etc.
2020-06-16 16:55:01 +00:00
Matt Keeler 64262d22d6 Make the Agent Cache more Context aware (#8092)
Blocking queries issues will still be uncancellable (that cannot be helped until we get rid of net/rpc). However this makes it so that if calling getWithIndex (like during a cache Notify go routine) we can cancell the outer routine. Previously it would keep issuing more blocking queries until the result state actually changed.
2020-06-15 15:43:32 +00:00
Freddy 2af14433be Merge pull request #8099 from hashicorp/gateway-services-endpoint 2020-06-12 21:15:25 +00:00
Freddy c9dbb6c51a Only pass one hostname via EDS and prefer healthy ones (#8084)
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>

Currently when passing hostname clusters to Envoy, we set each service instance registered with Consul as an LbEndpoint for the cluster.

However, Envoy can only handle one per cluster:
[2020-06-04 18:32:34.094][1][warning][config] [source/common/config/grpc_subscription_impl.cc:87] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) dc2.internal.ddd90499-9b47-91c5-4616-c0cbf0fc358a.consul: LOGICAL_DNS clusters must have a single locality_lb_endpoint and a single lb_endpoint, server.dc2.consul: LOGICAL_DNS clusters must have a single locality_lb_endpoint and a single lb_endpoint

Envoy is currently handling this gracefully by only picking one of the endpoints. However, we should avoid passing multiple to avoid these warning logs.

This PR:

* Ensures we only pass one endpoint, which is tied to one service instance.
* We prefer sending an endpoint which is marked as Healthy by Consul.
* If no endpoints are healthy we emit a warning and skip the cluster.
* If multiple unique hostnames are spread across service instances we emit a warning and let the user know which will be resolved.
2020-06-12 19:46:51 +00:00
Chris Piraino 7f89ab990e Allow users to set hosts to the wildcard specifier when TLS is disabled (#8083)
This allows easier demoing/testing of ingress gateways, while still
preserving the validation we have for DNSSANs
2020-06-11 15:03:46 +00:00
Chris Piraino 42c8f34788 Move ingress param to a new endpoint (#8081)
In discussion with team, it was pointed out that query parameters tend
to be filter mechanism, and that semantically the "/v1/health/connect"
endpoint should return "all healthy connect-enabled endpoints (e.g.
could be side car proxies or native instances) for this service so I can
connect with mTLS".

That does not fit an ingress gateway, so we remove the query parameter
and add a new endpoint "/v1/health/ingress" that semantically means
"all the healthy ingress gateway instances that I can connect to
to access this connect-enabled service without mTLS"
2020-06-10 18:07:41 +00:00
Chris Piraino ea1b54a826 Merge pull request #8064 from hashicorp/ingress/health-query-param
Add API query parameter ?ingress to allow users to find ingress gateways associated to a service
2020-06-09 21:09:09 +00:00
Hans Hasselberg a678b47c73 acl: do not resolve local tokens from remote dcs (#8068) 2020-06-09 19:14:19 +00:00
Kyle Havlovitz 9e6718ad0f Merge pull request #8040 from hashicorp/ingress/expose-cli
Ingress expose CLI command
2020-06-09 19:11:51 +00:00
Daniel Nephin 1bfb7f3b07 Merge pull request #7964 from hashicorp/dnephin/remove-patch-slice-of-maps-forward-compat
config: Use HookWeakDecodeFromSlice in place of PatchSliceOfMaps
2020-06-08 23:53:31 +00:00
Hans Hasselberg cfc95732f3
Tokens converted from legacy ACLs get their Hash computed (#8047) (#8054)
This allows new style token replication to work for legacy tokens as well when they change.
Fixes #5606
2020-06-08 23:36:55 +02:00
Hans Hasselberg b4f33b52a2 agent: add option to disable agent cache for HTTP endpoints (#8023)
This allows the operator to disable agent caching for the http endpoint.
It is on by default for backwards compatibility and if disabled will
ignore the url parameter `cached`.
2020-06-08 22:49:33 +02:00
Chris Piraino 165a9af053 Always require Host header values for http services (#7990)
Previously, we did not require the 'service-name.*' host header value
when on a single http service was exposed. However, this allows a user
to get into a situation where, if they add another service to the
listener, suddenly the previous service's traffic might not be routed
correctly. Thus, we always require the Host header, even if there is
only 1 service.

Also, we add the make the default domain matching more restrictive by
matching "service-name.ingress.*" by default. This lines up better with
the namespace case and more accurately matches the Consul DNS value we
expect people to use in this case.
2020-06-08 18:16:48 +00:00
Hans Hasselberg c675166e1b Setup intermediate_pki_path on secondary when using vault (#8001)
Make sure to mount vault backend for intermediate_pki_path on secondary
dc.
2020-06-05 19:37:21 +00:00
Hans Hasselberg de3e68c577 Merge pull request #7966 from hashicorp/pool_improvements
Agent connection pool cleanup
2020-06-05 19:03:24 +00:00
R.B. Boyer 89fc98322e tests: ensure that the ServiceExists helper function normalizes entmeta (#8025)
This fixes a unit test failure over in enterprise due to https://github.com/hashicorp/consul/pull/7384
2020-06-05 08:42:35 +00:00
Hans Hasselberg 0491a9301b tests: use constructor instead init (#8024) 2020-06-04 23:12:44 +02:00
R.B. Boyer ebc5fc039f server: don't activate federation state replication or anti-entropy until all servers are running 1.8.0+ (#8014) 2020-06-04 21:05:49 +00:00
Pierre Souchay 621862606e checks: when a service does not exists in an alias, consider it failing (#7384)
In current implementation of Consul, check alias cannot determine
if a service exists or not. Because a service without any check
is semantically considered as passing, so when no healthchecks
are found for an agent, the check was considered as passing.

But this make little sense as the current implementation does not
make any difference between:
 * a non-existing service (passing)
 * a service without any check (passing as well)

In order to make it work, we have to ensure that when a check did
not find any healthcheck, the service does indeed exists. If it
does not, lets consider the check as failing.
2020-06-04 12:51:23 +00:00
Freddy 5d2475232a Enable gateways to resolve hostnames to IPv4 addresses (#7999)
The DNS resolution will be handled by Envoy and defaults to LOGICAL_DNS. This discovery type can be overridden on a per-gateway basis with the envoy_dns_discovery_type Gateway Option.

If a service contains an instance with a hostname as an address we set the Envoy cluster to use DNS as the discovery type rather than EDS. Since both mesh gateways and terminating gateways route to clusters using SNI, whenever there is a mix of hostnames and IP addresses associated with a service we use the hostname + CDS rather than the IPs + EDS.

Note that we detect hostnames by attempting to parse the service instance's address as an IP. If it is not a valid IP we assume it is a hostname.
2020-06-03 18:51:33 -06:00
Matt Keeler 1e2754d59c Fix legacy management tokens in unupgraded secondary dcs (#7908)
The ACL.GetPolicy RPC endpoint was supposed to return the “parent” policy and not always the default policy. In the case of legacy management tokens the parent policy was supposed to be “manage”. The result of us not sending this properly was that operations that required specifically a management token such as saving a snapshot would not work in secondary DCs until they were upgraded.
2020-06-03 15:42:57 +00:00
Matt Keeler a539c5de88 Fix segfault due to race condition for checking server versions (#7957)
The ACL monitoring routine uses c.routers to check for server version updates. Therefore it needs to be started after initializing the routers.
2020-06-03 14:37:10 +00:00
R.B. Boyer 5404155d36 acl: allow auth methods created in the primary datacenter to optionally create global tokens (#7899) 2020-06-01 16:45:22 +00:00
R.B. Boyer c4b875cae4 acl: remove the deprecated `acl_enforce_version_8` option (#7991)
Fixes #7292
2020-06-01 10:40:22 -05:00
Jono Sosulska cedcbf3299 Replace whitelist/blacklist terminology with allowlist/denylist (#7971)
* Replace whitelist/blacklist terminology with allowlist/denylist
2020-06-01 10:40:14 -05:00
Daniel Nephin 1664067943 ci: Add staticcheck and fix most errors
Three of the checks are temporarily disabled to limit the size of the
diff, and allow us to enable all the other checks in CI.

In a follow up we can fix the issues reported by the other checks one
at a time, and enable them.
2020-06-01 10:40:04 -05:00
Daniel Nephin 1aeede5eb7 config: use the new HookTranslateKeys instead of lib.TranslateKeys
With the exception of CA provider config, which will be migrated at some
later time.
2020-06-01 10:39:58 -05:00
Daniel Nephin b11a615f0c Add alias struct tags for new decode hook 2020-06-01 10:39:51 -05:00
Raphaël Rondeau b29c954480 connect: fix endpoints clusterName when using cluster escape hatch (#7319)
```changelog
* fix(connect): fix endpoints clusterName when using cluster escape hatch
```
2020-06-01 10:35:31 -05:00
Pierre Souchay 0d86e802be Stop all watches before shuting down anything dring shutdown. (#7526)
This will prevent watches from being triggered.

```changelog
* fix(agent):  stop all watches before shuting down
```
2020-06-01 10:35:14 -05:00
Pierre Souchay 66612e5dc6 tests: added unit test to ensure watches are not re-triggered on consul reload (#7449)
This ensures no regression about https://github.com/hashicorp/consul/issues/7318
And ensure that https://github.com/hashicorp/consul/issues/7446 cannot happen anymore
2020-06-01 10:33:31 -05:00
Pierre Souchay 876ee89d4a Allow to restrict servers that can join a given Serf Consul cluster. (#7628)
Based on work done in https://github.com/hashicorp/memberlist/pull/196
this allows to restrict the IP ranges that can join a given Serf cluster
and be a member of the cluster.

Restrictions on IPs can be done separatly using 2 new differents flags
and config options to restrict IPs for LAN and WAN Serf.
2020-06-01 10:31:32 -05:00
R.B. Boyer c2b903b597 create lib/stringslice package (#7934) 2020-05-27 16:48:01 +00:00
R.B. Boyer b527e77850 agent: handle re-bootstrapping in a secondary datacenter when WAN federation via mesh gateways is configured (#7931)
The main fix here is to always union the `primary-gateways` list with
the list of mesh gateways in the primary returned from the replicated
federation states list. This will allow any replicated (incorrect) state
to be supplemented with user-configured (correct) state in the config
file. Eventually the game of random selection whack-a-mole will pick a
winning entry and re-replicate the latest federation states from the
primary. If the user-configured state is actually the incorrect one,
then the same eventual correct selection process will work in that case,
too.

The secondary fix is actually to finish making wanfed-via-mgws actually
work as originally designed. Once a secondary datacenter has replicated
federation states for the primary AND managed to stand up its own local
mesh gateways then all of the RPCs from a secondary to the primary
SHOULD go through two sets of mesh gateways to arrive in the consul
servers in the primary (one hop for the secondary datacenter's mesh
gateway, and one hop through the primary datacenter's mesh gateway).
This was neglected in the initial implementation. While everything
works, ideally we should treat communications that go around the mesh
gateways as just provided for bootstrapping purposes.

Now we heuristically use the success/failure history of the federation
state replicator goroutine loop to determine if our current mesh gateway
route is working as intended. If it is, we try using the local gateways,
and if those don't work we fall back on trying the primary via the union
of the replicated state and the go-discover configuration flags.

This can be improved slightly in the future by possibly initializing the
gateway choice to local on startup if we already have replicated state.
This PR does not address that improvement.

Fixes #7339
2020-05-27 16:32:22 +00:00
R.B. Boyer 1765fa854e connect: ensure proxy-defaults protocol is used for upstreams (#7938) 2020-05-21 21:09:51 +00:00
hashicorp-ci 7dd0a87286
update bindata_assetfs.go 2020-05-21 19:33:58 +00:00
Daniel Nephin 7925a0074c Merge pull request #7933 from hashicorp/dnephin/state-txn-missing-errors
state: fix unhandled error
2020-05-21 17:03:33 +00:00
Aleksandr Zagaevskiy 6aecf89418 Preserve ModifyIndex for unchanged entry in KVS TXN (#7832) 2020-05-21 17:03:16 +00:00
Seth Hoenig 352ed2c13b grpc: use default resolver scheme for grpc dialing (#7617)
Currently checks of type gRPC will emit log messages such as,

    2020/02/12 13:48:22 [INFO] parsed scheme: ""
    2020/02/12 13:48:22 [INFO] scheme "" not registered, fallback to default scheme

Without adding full support for using custom gRPC schemes (maybe that's
right long-term path) we can just supply the default scheme as provided
by the grpc library.

Fixes https://github.com/hashicorp/consul/issues/7274
and https://github.com/hashicorp/nomad/issues/7415
2020-05-21 17:01:47 +00:00
Daniel Nephin c02d4e1390 Merge pull request #7894 from hashicorp/dnephin/add-linter-staticcheck-1
Fix some bugs/issues found by staticcheck
2020-05-21 17:01:15 +00:00
Kyle Havlovitz 0bcbed16ca Standardize support for Tagged and BindAddresses in Ingress Gateways (#7924)
* Standardize support for Tagged and BindAddresses in Ingress Gateways

This updates the TaggedAddresses and BindAddresses behavior for Ingress
to match Mesh/Terminating gateways. The `consul connect envoy` command
now also allows passing an address without a port for tagged/bind
addresses.

* Update command/connect/envoy/envoy.go

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* PR comments

* Check to see if address is an actual IP address

* Update agent/xds/listeners.go

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* fix whitespace

Co-authored-by: Chris Piraino <cpiraino@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2020-05-21 14:08:43 +00:00
Chris Piraino 6969d08361 Merge pull request #7898 from hashicorp/bug/update-gateways-on-config-entry-delete
Remove error from GatewayServices RPC when a service is not a gateway
2020-05-18 18:03:35 +00:00
hashicorp-ci b1c9c5c571
update bindata_assetfs.go 2020-05-14 14:33:09 +00:00
Pierre Souchay cf55e81c06
tests: fix unstable test `TestAgentAntiEntropy_Checks`. (#7594)
Example of failure: https://circleci.com/gh/hashicorp/consul/153932#tests/containers/2
2020-05-14 09:54:49 +02:00
Kit Patella ad1d4d4d07 http: migrate from instrumentation in s.wrap() to an s.enterpriseHandler() 2020-05-13 15:47:05 -07:00
Matt Keeler acccdbe45c
Fix identity resolution on clients and in secondary dcs (#7862)
Previously this happened to be using the method on the Server/Client that was meant to allow the ACLResolver to locally resolve tokens. On Servers that had tokens (primary or secondary dc + token replication) this function would lookup the token from raft and return the ACLIdentity. On clients this was always a noop. We inadvertently used this function instead of creating a new one when we added logging accessor ids for permission denied RPC requests. 

With this commit, a new method is used for resolving the identity properly via the ACLResolver which may still resolve locally in the case of being on a server with tokens but also supports remote token resolution.
2020-05-13 13:00:08 -04:00
Chris Piraino 7a7760bfd5
Make new gateway tests compatible with enterprise (#7856) 2020-05-12 13:48:20 -05:00
Daniel Nephin 600645b5f9 Add unconvert linter
To find unnecessary type convertions
2020-05-12 13:47:25 -04:00
Drew Bailey c9d0b83277 Value is already an int, remove type cast 2020-05-12 13:13:09 -04:00
Daniel Nephin 9d5ab443a7
Merge pull request #7689 from hashicorp/dnephin/remove-deadcode-1
Remove some dead code
2020-05-12 12:33:59 -04:00
Daniel Nephin 47238a693d
Merge pull request #7819 from hashicorp/dnephin/remove-t.Parallel-1
test: Remove t.Parallel() from agent/structs tests
2020-05-12 12:11:57 -04:00
R.B. Boyer 1efafd7523
acl: add auth method for JWTs (#7846) 2020-05-11 20:59:29 -05:00
Kit Patella 58ee349a83
Merge pull request #7843 from hashicorp/oss-sync/auditing-config
agent/config: Fix tests & include Audit struct as a pointer on Config
2020-05-11 14:23:44 -07:00
Kit Patella 10b3478a4d agent/config: include Audit struct as a pointer on Config, fix tests 2020-05-11 14:13:05 -07:00
Kit Patella b5564751bf
Merge pull request #7841 from hashicorp/oss-sync/auditing-config
OSS sync - Auditing config
2020-05-11 13:44:38 -07:00
Kit Patella f5030957d0 agent/config: add auditing config to OSS and add to enterpriseConfigMap exclusions 2020-05-11 13:27:35 -07:00
Chris Piraino c21052457b
Return early from updateGatewayServices if nothing to update (#7838)
* Return early from updateGatewayServices if nothing to update

Previously, we returned an empty slice of gatewayServices, which caused
us to accidentally delete everything in the memdb table

* PR comment and better formatting
2020-05-11 14:46:48 -05:00
Chris Piraino 4d6751bf16
Fix TestInternal_GatewayServiceDump_Ingress (#7840)
Protocol was added as a field on GatewayServices after
GatewayServiceDump PR branch was created.
2020-05-11 14:46:31 -05:00
R.B. Boyer 7414a3fa53
cli: ensure 'acl auth-method update' doesn't deep merge the Config field (#7839) 2020-05-11 14:21:17 -05:00
Chris Piraino 74c0543ef2 PR comment and better formatting 2020-05-11 14:04:59 -05:00
Chris Piraino fb9ee9d892 Return early from updateGatewayServices if nothing to update
Previously, we returned an empty slice of gatewayServices, which caused
us to accidentally delete everything in the memdb table
2020-05-11 12:38:04 -05:00
Freddy b3ec383d04
Gateway Services Nodes UI Endpoint (#7685)
The endpoint supports queries for both Ingress Gateways and Terminating Gateways. Used to display a gateway's linked services in the UI.
2020-05-11 11:35:17 -06:00
Kyle Havlovitz 136549205c
Merge pull request #7759 from hashicorp/ingress/tls-hosts
Add TLS option for Ingress Gateway listeners
2020-05-11 09:18:43 -07:00
Kyle Havlovitz 8d140ce9af Disallow the blanket wildcard prefix from being used as custom host 2020-05-08 20:24:18 -07:00
Chris Piraino a0e1f57ac2 Remove development log line 2020-05-08 20:24:18 -07:00
Chris Piraino 26f92e74f6 Compute all valid DNSSANs for ingress gateways
For DNSSANs we take into account the following and compute the
appropriate wildcard values:
- source datacenter
- namespaces
- alt domains
2020-05-08 20:23:17 -07:00
Daniel Nephin 5655d7f34e Add outlier_detection check to integration test
Fix decoding of time.Duration types.
2020-05-08 14:56:57 -04:00
Daniel Nephin eaa05d623a xds: Add passive health check config for upstreams 2020-05-08 14:56:57 -04:00
Chris Piraino 429d0cedd2
Restoring config entries updates the gateway-services table (#7811)
- Adds a new validateConfigEntryEnterprise function
- Also fixes some state store tests that were failing in enterprise
2020-05-08 13:24:33 -05:00
Daniel Nephin e60bb9f102 test: Remove t.Parallel() from agent/structs tests
go test will only run tests in parallel within a single package. In this case the package test run time is exactly the same with or without t.Parallel() (~0.7s).

In generally we should avoid t.Parallel() as it causes a number of problems with `go test` not reporting failure messages correctly. I encountered one of these problems, which is what prompted this change.  Since `t.Parallel` is not providing any benefit in this package, this commit removes it.

The change was automated with:

    git grep -l 't.Parallel' | xargs sed -i -e '/t.Parallel/d'
2020-05-08 14:06:10 -04:00
Freddy c32a4f1ece
Fix up enterprise compatibility for gateways (#7813) 2020-05-08 09:44:34 -06:00
Jono Sosulska 9b363e9f23
Fix spelling of deregister (#7804) 2020-05-08 10:03:45 -04:00
Chris Piraino f55e20a2f7
Allow ingress gateways to send empty clusters, routes, and listeners (#7795)
This is useful when updating an config entry with no services, and the
expected behavior is that envoy closes all listeners and clusters.

We also allow empty routes because ingress gateways name route
configurations based on the port of the listener, so it is important we
remove any stale routes. Then, if a new listener with an old port is
added, we will not have to deal with stale routes hanging around routing
to the wrong place.

Endpoints are associated with clusters, and thus by deleting the
clusters we don't have to care about sending empty endpoint responses.
2020-05-07 16:19:25 -05:00
Chris Piraino 0bd5618cb2 Cleanup proxycfg for TLS
- Use correct enterprise metadata for finding config entry
- nil out cancel functions on config snapshot copy
- Look at HostsSet when checking validity
2020-05-07 10:22:57 -05:00
Chris Piraino 5105bf3d67
Require individual services in ingress entry to match protocols (#7774)
We require any non-wildcard services to match the protocol defined in
the listener on write, so that we can maintain a consistent experience
through ingress gateways. This also helps guard against accidental
misconfiguration by a user.

- Update tests that require an updated protocol for ingress gateways
2020-05-06 16:09:24 -05:00
Freddy b069887b2a
Remove timeout and call to Fatal from goroutine (#7797) 2020-05-06 14:33:17 -06:00
Chris Piraino 0c22eacca8 Add TLS field to ingress API structs
- Adds test in api and command/config/write packages
2020-05-06 15:12:02 -05:00
Chris Piraino 30792e933b Add test for adding DNSSAN for ConnectCALeaf cache type 2020-05-06 15:12:02 -05:00
Chris Piraino 0b9ba9660d Validate hosts input in ingress gateway config entry
We can only allow host names that are valid domain names because we put
these hosts into a DNSSAN. In addition, we validate that the wildcard
specifier '*' is only present as the leftmost label to allow for a
wildcard DNSSAN and associated wildcard Host routing in the ingress
gateway proxy.
2020-05-06 15:12:02 -05:00
Kyle Havlovitz f14c54e25e Add TLS option and DNS SAN support to ingress config
xds: Only set TLS context for ingress listener when requested
2020-05-06 15:12:02 -05:00
Chris Piraino 905279f5d1 A proxy-default config entry only exists in the default namespace 2020-05-06 15:06:14 -05:00
Chris Piraino d498a0afc9 Correctly set a namespace label in the required domain for xds routes
If an upstream is not in the default namespace, we expect DNS requests
to be served over "<service-name>.ingress.<namespace>.*"
2020-05-06 15:06:14 -05:00
Chris Piraino 114a18e890 Remove outdated comment 2020-05-06 15:06:14 -05:00
Chris Piraino d8517bd6fd Better document wildcard specifier interactions 2020-05-06 15:06:14 -05:00
Chris Piraino 45e635286a Re-add comment on connect-proxy virtual hosts 2020-05-06 15:06:14 -05:00
Kyle Havlovitz f9672f9bf1 Make sure IngressHosts isn't parsed during JSON decode 2020-05-06 15:06:14 -05:00
Chris Piraino c44f877758 Comment why it is ok to expect upstreams slice to not be empty 2020-05-06 15:06:13 -05:00
Chris Piraino 881760f701 xds: Use only the port number as the configured route name
This removes duplication of protocol from the stats_prefix
2020-05-06 15:06:13 -05:00
Kyle Havlovitz 89e6b16815 Filter wildcard gateway services to match listener protocol
This now requires some type of protocol setting in ingress gateway tests
to ensure the services are not filtered out.

- small refactor to add a max(x, y) function
- Use internal configEntryTxn function and add MaxUint64 to lib
2020-05-06 15:06:13 -05:00
Chris Piraino f40833d094 Allow Hosts field to be set on an ingress config entry
- Validate that this cannot be set on a 'tcp' listener nor on a wildcard
service.
- Add Hosts field to api and test in consul config write CLI
- xds: Configure envoy with user-provided hosts from ingress gateways
2020-05-06 15:06:13 -05:00
Chris Piraino b73a13fc9e Remove service_subset field from ingress config entry
We decided that this was not a useful MVP feature, and just added
unnecessary complexity
2020-05-06 15:06:13 -05:00
Kyle Havlovitz 711d1389aa Support multiple listeners referencing the same service in gateway definitions 2020-05-06 15:06:13 -05:00
Kyle Havlovitz 247f9eaf13 Allow ingress gateways to route traffic based on Host header
This commit adds the necessary changes to allow an ingress gateway to
route traffic from a single defined port to multiple different upstream
services in the Consul mesh.

To do this, we now require all HTTP requests coming into the ingress
gateway to specify a Host header that matches "<service-name>.*" in
order to correctly route traffic to the correct service.

- Differentiate multiple listener's route names by port
- Adds a case in xds for allowing default discovery chains to create a
  route configuration when on an ingress gateway. This allows default
  services to easily use host header routing
- ingress-gateways have a single route config for each listener
  that utilizes domain matching to route to different services.
2020-05-06 15:06:13 -05:00
R.B. Boyer a854e4d9c5
acl: oss plumbing to support auth method namespace rules in enterprise (#7794)
This includes website docs updates.
2020-05-06 13:48:04 -05:00
R.B. Boyer 3242d0816d
test: make the kube auth method test helper use freeport (#7788) 2020-05-05 16:55:21 -05:00
Hans Hasselberg 096a2f2f02 network_segments: stop advertising segment tags 2020-05-05 21:32:05 +02:00
Hans Hasselberg 995a24b8e4 agent: refactor to use a single addrFn 2020-05-05 21:08:10 +02:00
Hans Hasselberg 6994c0d47f agent: rename local/global to src/dst 2020-05-05 21:07:34 +02:00
Chris Piraino 69b44fb942
Construct a default destination if one does not exist for service-router (#7783) 2020-05-05 10:49:50 -05:00
R.B. Boyer 22eb016153
acl: add MaxTokenTTL field to auth methods (#7779)
When set to a non zero value it will limit the ExpirationTime of all
tokens created via the auth method.
2020-05-04 17:02:57 -05:00
R.B. Boyer ca52ba7068
acl: add DisplayName field to auth methods (#7769)
Also add a few missing acl fields in the api.
2020-05-04 15:18:25 -05:00
Hans Hasselberg c4093c87cc
agent: don't let left nodes hold onto their node-id (#7747) 2020-05-04 18:39:08 +02:00
Matt Keeler daec810e34
Merge pull request #7714 from hashicorp/oss-sync/msp-agent-token 2020-05-04 11:33:50 -04:00
Matt Keeler cbe3a70f56
Update enterprise configurations to be in OSS
This will emit warnings about the configs not doing anything but still allow them to be parsed.

This also added the warnings for enterprise fields that we already had in OSS but didn’t change their enforcement behavior. For example, attempting to use a network segment will cause a hard error in OSS.
2020-05-04 10:21:05 -04:00
R.B. Boyer 9533451a63
acl: refactor the authmethod.Validator interface (#7760)
This is a collection of refactors that make upcoming PRs easier to digest.

The main change is the introduction of the authmethod.Identity struct.
In the one and only current auth method (type=kubernetes) all of the
trusted identity attributes are both selectable and projectable, so they
were just passed around as a map[string]string.

When namespaces were added, this was slightly changed so that the
enterprise metadata can also come back from the login operation, so
login now returned two fields.

Now with some upcoming auth methods it won't be true that all identity
attributes will be both selectable and projectable, so rather than
update the login function to return 3 pieces of data it seemed worth it
to wrap those fields up and give them a proper name.
2020-05-01 17:35:28 -05:00
R.B. Boyer 54ba8e3868
acl: change authmethod.Validator to take a logger (#7758) 2020-05-01 15:55:26 -05:00
R.B. Boyer 8927b54121
test: move some test helpers over from enterprise (#7754) 2020-05-01 14:52:15 -05:00
R.B. Boyer b282268408
sdk: extracting testutil.RequireErrorContains from various places it was duplicated (#7753) 2020-05-01 11:56:34 -05:00
Hans Hasselberg 51549bd232
rpc: oss changes for network area connection pooling (#7735) 2020-04-30 22:12:17 +02:00
Freddy 021f0ee36e
Watch fallback channel for gateways that do not exist (#7715)
Also ensure that WatchSets in tests are reset between calls to watchFired. 
Any time a watch fires, subsequent calls to watchFired on the same WatchSet
will also return true even if there were no changes.
2020-04-29 16:52:27 -06:00