Commit Graph

4944 Commits

Author SHA1 Message Date
Freddy d9fe3578ac
Merge pull request #14734 from hashicorp/NET-643-update-mesh-gateway-envoy-config-for-inbound-peering-control-plane-traffic 2022-10-03 12:54:11 -06:00
freddygv b15d41534f Update xds generation for peering over mesh gws
This commit adds the xDS resources needed for INBOUND traffic from peer
clusters:

- 1 filter chain for all inbound peering requests.
- 1 cluster for all inbound peering requests.
- 1 endpoint per voting server with the gRPC TLS port configured.

There is one filter chain and cluster because unlike with WAN
federation, peer clusters will not attempt to dial individual servers.
Peer clusters will only dial the local mesh gateway addresses.
2022-10-03 12:42:27 -06:00
freddygv a8c4d6bc55 Share mgw addrs in peering stream if needed
This commit adds handling so that the replication stream considers
whether the user intends to peer through mesh gateways.

The subscription will return server or mesh gateway addresses depending
on the mesh configuration setting. These watches can be updated at
runtime by modifying the mesh config entry.
2022-10-03 11:42:20 -06:00
freddygv 4ff9d475b0 Return mesh gateway addrs if peering through mgw 2022-10-03 11:35:10 -06:00
chappie ad7295e5d9
Merge pull request #14811 from hashicorp/chappie/dns
Add DNS gRPC proxying support
2022-10-03 08:02:48 -07:00
Chris Chapman d7b5351b66
Making suggested comments 2022-09-30 15:03:33 -07:00
Chris Chapman 46bea72212
Making suggested changes 2022-09-30 14:51:12 -07:00
Chris Chapman a05563b788
Update comment 2022-09-30 09:35:01 -07:00
DanStough 7f8971d77f chore: fix flakey scada provider test 2022-09-30 11:56:40 -04:00
Chris Chapman 81e267171b
Bind a dns mux handler to gRPC proxy 2022-09-29 21:44:45 -07:00
Chris Chapman 7bc9cad180
Adding grpc handler for dns proxy 2022-09-29 21:19:51 -07:00
Eric Haberkorn 80e51ff907
Add exported services event to cluster peering replication. (#14797) 2022-09-29 15:37:19 -04:00
Ashwin Venkatesh 4ba260958c
bug: watch local mesh gateways in non-default partitions with agentless (#14799) 2022-09-29 13:19:04 -04:00
cskh 69f40df548
feat(ingress gateway: support configuring limits in ingress-gateway c… (#14749)
* feat(ingress gateway: support configuring limits in ingress-gateway config entry

- a new Defaults field with max_connections, max_pending_connections, max_requests
  is added to ingress gateway config entry
- new field max_connections, max_pending_connections, max_requests in
  individual services to overwrite the value in Default
- added unit test and integration test
- updated doc

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
2022-09-28 14:56:46 -04:00
malizz 84b0f408fa
Support Stale Queries for Trust Bundle Lookups (#14724)
* initial commit

* add tags, add conversations

* add test for query options utility functions

* update previous tests

* fix test

* don't error out on empty context

* add changelog

* update decode config
2022-09-28 09:56:59 -07:00
Eric Haberkorn 6570d5f004
Enable outbound peered requests to go through local mesh gateway (#14763) 2022-09-27 09:49:28 -04:00
Nick Ethier 1c1b0994b8
add HCP integration component (#14723)
* add HCP integration

* lint: use non-deprecated logging interface
2022-09-26 14:58:15 -04:00
Derek Menteer aa4709ab74
Add envoy connection balancing. (#14616)
Add envoy connection balancing config.
2022-09-26 11:29:06 -05:00
Chris S. Kim 2203cdc4db Add new internal endpoint to list exported services to a peer 2022-09-23 09:43:56 -04:00
freddygv d818d7b096 Manage local server watches depending on mesh cfg
Routing peering control plane traffic through mesh gateways can be
enabled or disabled at runtime with the mesh config entry.

This commit updates proxycfg to add or cancel watches for local servers
depending on this central config.

Note that WAN federation over mesh gateways is determined by a service
metadata flag, and any updates to the gateway service registration will
force the creation of a new snapshot. If enabled, WAN-fed over mesh
gateways will trigger a local server watch on initialize().

Because of this we will only add/remove server watches if WAN federation
over mesh gateways is disabled.
2022-09-22 19:32:10 -06:00
Alessandro De Blasis 461b42ed48 fix(check): added missing OSService props 2022-09-21 13:10:21 +01:00
Alessandro De Blasis 5719fd6560 fix(checks): os_service OK message in output 2022-09-21 09:27:33 +01:00
Alessandro De Blasis f440966a38 fix(checks): os_service lifecycle bugfix 2022-09-21 09:26:47 +01:00
Alessandro De Blasis fc0dd92dcf fix(agent): uninitialized map panic error 2022-09-21 09:25:54 +01:00
malizz 1a0aa38a82
increase the size of txn to support vault (#14599)
* increase the size of txn to support vault

* add test, revert change to acl endpoint

* add changelog

* update test, add passing test case

* Update .changelog/14599.txt

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2022-09-19 09:07:19 -07:00
freddygv 5fbb26525b Add awareness of server mode to TLS configurator
Preivously the TLS configurator would default to presenting auto TLS
certificates as client certificates.

Server agents should not have this behavior and should instead present
the manually configured certs. The autoTLS certs for servers are
exclusively used for peering and should not be used as the default for
outbound communication.
2022-09-16 17:57:10 -06:00
freddygv f30bc96239 Test fixes
- Pulls in CLI test fix from main
- Updates psutils to fix TestAgent_Host on M1 Mac
2022-09-16 17:57:10 -06:00
freddygv 02d3ce1039 Add server certificate manager
This certificate manager will request a leaf certificate for server
agents and then keep them up to date.
2022-09-16 17:57:10 -06:00
freddygv 0e5131bd33 Generate ACL token for server management
This commit introduces a new ACL token used for internal server
management purposes.

It has a few key properties:
- It has unlimited permissions.
- It is persisted through Raft as System Metadata rather than in the
ACL tokens table. This is to avoid users seeing or modifying it.
- It is re-generated on leadership establishment.
2022-09-16 17:54:34 -06:00
freddygv 0ea3353537 Add handling in agent cache for server leaf certs 2022-09-16 17:54:34 -06:00
Kyle Havlovitz 0d9ae52643
Merge pull request #14598 from hashicorp/root-removal-fix
connect/ca: Don't discard old roots on primaryInitialize
2022-09-15 14:36:01 -07:00
Kyle Havlovitz 6105a7fd9f connect/ca: don't discard old roots on primaryInitialize 2022-09-15 12:59:09 -07:00
Gabriel Santos e53af28bd7
Middleware: `RequestRecorder` reports calls below 1ms as decimal value (#12905)
* Typos

* Test failing

* Convert values <1ms to decimal

* Fix test

* Update docs and test error msg

* Applied suggested changes to test case

* Changelog file and suggested changes

* Update .changelog/12905.txt

Co-authored-by: Chris S. Kim <kisunji92@gmail.com>

* suggested change - start duration with microseconds instead of nanoseconds

* fix error

* suggested change - floats

Co-authored-by: alex <8968914+acpana@users.noreply.github.com>
Co-authored-by: Chris S. Kim <kisunji92@gmail.com>
2022-09-15 13:04:37 -04:00
Daniel Graña 8c98172f53
[BUGFIX] Do not use interval as timeout (#14619)
Do not use interval as timeout
2022-09-15 12:39:48 -04:00
Evan Culver d0416f593c
connect: Bump latest Envoy to 1.23.1 in test matrix (#14573) 2022-09-14 13:20:16 -07:00
DanStough 485e1b5d4e fix(peering): generate token metrics only for leader 2022-09-14 11:37:30 -04:00
DanStough 2a2debee64 feat(peering): validate server name conflicts on establish 2022-09-14 11:37:30 -04:00
Kyle Havlovitz 60cee76746
Merge pull request #14516 from hashicorp/ca-ttl-fixes
Fix inconsistent TTL behavior in CA providers
2022-09-13 16:07:36 -07:00
Kyle Havlovitz d67bccd210 Update intermediate pki mount/role when reconfiguring Vault provider 2022-09-13 15:42:26 -07:00
Kyle Havlovitz f46955101a connect/ca: Clarify behavior around IntermediateCertTTL in CA config 2022-09-13 15:42:26 -07:00
DanStough 0150e88200 feat: add PeerThroughMeshGateways to mesh config 2022-09-13 17:19:54 -04:00
Derek Menteer 0aa13733a0
Add CSR check for number of URIs. (#14579)
Add CSR check for number of URIs.
2022-09-13 14:21:47 -05:00
Derek Menteer db83ff4fa6 Add input validation for auto-config JWT authorization checks. 2022-09-13 11:16:36 -05:00
cskh f22685b969
Config-entry: Support proxy config in service-defaults (#14395)
* Config-entry: Support proxy config in service-defaults

* Update website/content/docs/connect/config-entries/service-defaults.mdx

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
2022-09-12 10:41:58 -04:00
Eric Haberkorn aa8268e50c
Implement Cluster Peering Redirects (#14445)
implement cluster peering redirects
2022-09-09 13:58:28 -04:00
skpratt b761589340
add non-double-prefixed metrics (#14193) 2022-09-09 12:13:43 -05:00
skpratt 19f79aa9a6
PR #14057 follow up fix: service id parsing from sidecar id (#14541)
* fix service id parsing from sidecar id

* simplify suffix trimming
2022-09-09 09:47:10 -05:00
Dan Upton 1c2c975b0b
xDS Load Balancing (#14397)
Prior to #13244, connect proxies and gateways could only be configured by an
xDS session served by the local client agent.

In an upcoming release, it will be possible to deploy a Consul service mesh
without client agents. In this model, xDS sessions will be handled by the
servers themselves, which necessitates load-balancing to prevent a single
server from receiving a disproportionate amount of load and becoming
overwhelmed.

This introduces a simple form of load-balancing where Consul will attempt to
achieve an even spread of load (xDS sessions) between all healthy servers.
It does so by implementing a concurrent session limiter (limiter.SessionLimiter)
and adjusting the limit according to autopilot state and proxy service
registrations in the catalog.

If a server is already over capacity (i.e. the session limit is lowered),
Consul will begin draining sessions to rebalance the load. This will result
in the client receiving a `RESOURCE_EXHAUSTED` status code. It is the client's
responsibility to observe this response and reconnect to a different server.

Users of the gRPC client connection brokered by the
consul-server-connection-manager library will get this for free.

The rate at which Consul will drain sessions to rebalance load is scaled
dynamically based on the number of proxies in the catalog.
2022-09-09 15:02:01 +01:00
Derek Menteer f7c884f0af Merge branch 'main' of github.com:hashicorp/consul into derekm/split-grpc-ports 2022-09-08 14:53:08 -05:00
Derek Menteer bfe7c5e8af Remove rebuilding grpc server. 2022-09-08 13:45:44 -05:00
Derek Menteer 80d31458e5 Various cleanups. 2022-09-08 10:51:50 -05:00
Chris S. Kim 03df6c3ac6
Reuse http.DefaultTransport in UIMetricsProxy (#14521)
http.Transport keeps a pool of connections and should be reused when possible. We instantiate a new http.DefaultTransport for every metrics request, making large numbers of concurrent requests inefficiently spin up new connections instead of reusing open ones.
2022-09-08 11:02:05 -04:00
Chris S. Kim 1c4a6eef4f
Merge pull request #14285 from hashicorp/NET-638-push-server-address-updates-to-the-peer
peering: Subscribe to server address changes and push updates to peers
2022-09-07 09:30:45 -04:00
skpratt 3bf1edfb3f
move port and default check logic to locked step (#14057) 2022-09-06 19:35:31 -05:00
Freddy f4dfd42e0a
Add SpiffeID for Consul server agents (#14485)
Co-authored-by: Eric Haberkorn <erichaberkorn@gmail.com>

By adding a SpiffeID for server agents, servers can now request a leaf
certificate from the Connect CA.

This new Spiffe ID has a key property: servers are identified by their
datacenter name and trust domain. All servers that share these
attributes will share a ServerURI.

The aim is to use these certificates to verify the server name of ANY
server in a Consul datacenter.
2022-09-06 17:58:13 -06:00
Daniel Upton 8c46e48e0d proxycfg-glue: server-local implementation of IntentionUpstreamsDestination
This is the OSS portion of enterprise PR 2463.

Generalises the serverIntentionUpstreams type to support matching on a
service or destination.
2022-09-06 23:27:25 +01:00
Daniel Upton f8dba7e9ac proxycfg-glue: server-local implementation of InternalServiceDump
This is the OSS portion of enterprise PR 2489.

This PR introduces a server-local implementation of the
proxycfg.InternalServiceDump interface that sources data from a blocking query
against the server's state store.

For simplicity, it only implements the subset of the Internal.ServiceDump RPC
handler actually used by proxycfg - as such the result type has been changed
to IndexedCheckServiceNodes to avoid confusion.
2022-09-06 23:27:25 +01:00
Daniel Upton a31738f76f proxycfg-glue: server-local implementation of ResolvedServiceConfig
This is the OSS portion of enterprise PR 2460.

Introduces a server-local implementation of the proxycfg.ResolvedServiceConfig
interface that sources data from a blocking query against the server's state
store.

It moves the service config resolution logic into the agent/configentry package
so that it can be used in both the RPC handler and data source.

I've also done a little re-arranging and adding comments to call out data
sources for which there is to be no server-local equivalent.
2022-09-06 23:27:25 +01:00
Derek Menteer bf769daae4 Merge branch 'main' of github.com:hashicorp/consul into derekm/split-grpc-ports 2022-09-06 10:51:04 -05:00
Derek Menteer 02ae66bda8 Add kv txn get-not-exists operation. 2022-09-06 10:28:59 -05:00
Chris S. Kim 953808e899 PR feedback on terminated state checking 2022-09-06 10:28:20 -04:00
Chris S. Kim ddb9375cb6 Add testcase for parsing grpc_port 2022-09-06 10:17:44 -04:00
Kyle Havlovitz d97ccccdd5
Merge pull request #14429 from hashicorp/ca-prune-intermediates
Prune old expired intermediate certs when appending a new one
2022-09-02 15:34:33 -07:00
cskh 0f7d4efac3
fix(txn api): missing proxy config in registering proxy service (#14471)
* fix(txn api): missing proxy config in registering proxy service
2022-09-02 14:28:05 -04:00
Chris S. Kim ec36755cc0 Properly assert for ServerAddresses replication request 2022-09-02 11:44:54 -04:00
Chris S. Kim d1d9dbff8e Fix terminate not returning early 2022-09-02 11:44:38 -04:00
Derek Menteer f64771c707 Address PR comments. 2022-09-01 16:54:24 -05:00
Kyle Havlovitz 0c2fb7252d Prune intermediates before appending new one 2022-09-01 14:24:30 -07:00
Luke Kysow 81d7cc41dc
Use proxy address for default check (#14433)
When a sidecar proxy is registered, a check is automatically added.
Previously, the address this check used was the underlying service's
address instead of the proxy's address, even though the check is testing
if the proxy is up.

This worked in most cases because the proxy ran on the same IP as the
underlying service but it's not guaranteed and so the proper default
address should be the proxy's address.
2022-09-01 14:03:35 -07:00
malizz f1054dada9
fix TestProxyConfigEntry (#14435) 2022-09-01 11:37:47 -07:00
malizz b3ac8f48ca
Add additional parameters to envoy passive health check config (#14238)
* draft commit

* add changelog, update test

* remove extra param

* fix test

* update type to account for nil value

* add test for custom passive health check

* update comments and tests

* update description in docs

* fix missing commas
2022-09-01 09:59:11 -07:00
Chris S. Kim f2b147e575 Add Internal.ServiceDump support for querying by PeerName 2022-09-01 10:32:59 -04:00
Chris S. Kim e62f830fa8
Merge pull request #13998 from jorgemarey/f-new-tracing-envoy
Add new envoy tracing configuration
2022-09-01 08:57:23 -04:00
Derek Menteer cf7f24a6ec Change serf-tag references to field references. 2022-08-31 16:38:42 -05:00
malizz a80e0bcd00
validate args before deleting proxy defaults (#14290)
* validate args before deleting proxy defaults

* add changelog

* validate name when normalizing proxy defaults

* add test for proxyConfigEntry

* add comments
2022-08-31 13:03:38 -07:00
Kyle Havlovitz 113454645d Prune old expired intermediate certs when appending a new one 2022-08-31 11:41:58 -07:00
Alessandro De Blasis 60c7c831c6 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service 2022-08-30 18:49:20 +01:00
Eric Haberkorn 3726a0ab7a
Finish up cluster peering failover (#14396) 2022-08-30 11:46:34 -04:00
Chris S. Kim 560d410c6d Merge branch 'main' into NET-638-push-server-address-updates-to-the-peer
# Conflicts:
#	agent/grpc-external/services/peerstream/stream_test.go
2022-08-30 11:09:25 -04:00
Jorge Marey 3f3bb8831e Fix typos. Add test. Add documentation 2022-08-30 16:59:02 +02:00
Jorge Marey ed7b34128f Add new tracing configuration 2022-08-30 16:59:02 +02:00
Freddy 97d1db759f
Merge pull request #13496 from maxb/fix-kv_entries-metric 2022-08-29 15:35:11 -06:00
Freddy 829a2a8722
Merge pull request #14364 from hashicorp/peering/term-delete 2022-08-29 15:33:18 -06:00
Max Bowsher decc9231ee Merge branch 'main' into fix-kv_entries-metric 2022-08-29 22:22:10 +01:00
Chris S. Kim 5010fa5c03
Merge pull request #14371 from hashicorp/kisunji/peering-metrics-update
Adjust metrics reporting for peering tracker
2022-08-29 17:16:19 -04:00
Chris S. Kim 74ddf040dd Add heartbeat timeout grace period when accounting for peering health 2022-08-29 16:32:26 -04:00
Derek Menteer 0ceec9017b Expose `grpc_tls` via serf for cluster peering. 2022-08-29 13:43:49 -05:00
Derek Menteer 1255a8a20d Add separate grpc_tls port.
To ease the transition for users, the original gRPC
port can still operate in a deprecated mode as either
plain-text or TLS mode. This behavior should be removed
in a future release whenever we no longer support this.

The resulting behavior from this commit is:
  `ports.grpc > 0 && ports.grpc_tls > 0` spawns both plain-text and tls ports.
  `ports.grpc > 0 && grpc.tls == undefined` spawns a single plain-text port.
  `ports.grpc > 0 && grpc.tls != undefined` spawns a single tls port (backwards compat mode).
2022-08-29 13:43:43 -05:00
freddygv 310608fb19 Add validation to prevent switching dialing mode
This prevents unexpected changes to the output of ShouldDial, which
should never change unless a peering is deleted and recreated.
2022-08-29 12:31:13 -06:00
Eric Haberkorn 72f90754ae
Update max_ejection_percent on outlier detection for peered clusters to 100% (#14373)
We can't trust health checks on peered services when service resolvers,
splitters and routers are used.
2022-08-29 13:46:41 -04:00
Alessandro De Blasis 26cc56bc68 fix(agent): removed redundant code in docker check as well 2022-08-29 18:15:59 +01:00
Alessandro De Blasis c0d647d11e fix(agent): removed redundant check on prev. running check 2022-08-29 17:53:39 +01:00
Chris S. Kim def529edd3 Rename test 2022-08-29 10:34:50 -04:00
Chris S. Kim 93271f649c Fix test 2022-08-29 10:20:30 -04:00
Eric Haberkorn 1099665473
Update the structs and discovery chain for service resolver redirects to cluster peers. (#14366) 2022-08-29 09:51:32 -04:00
Alessandro De Blasis f3437eaf05 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service
Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2022-08-28 18:09:31 +01:00
Alessandro De Blasis f634e36811 fix(OSServiceCheck): fixes following code-review 2022-08-28 17:56:30 +01:00
Chris S. Kim 4d97e2f936 Adjust metrics reporting for peering tracker 2022-08-26 17:34:17 -04:00
freddygv 650e48624d Allow terminated peerings to be deleted
Peerings are terminated when a peer decides to delete the peering from
their end. Deleting a peering sends a termination message to the peer
and triggers them to mark the peering as terminated but does NOT delete
the peering itself. This is to prevent peerings from disappearing from
both sides just because one side deleted them.

Previously the Delete endpoint was skipping the deletion if the peering
was not marked as active. However, terminated peerings are also
inactive.

This PR makes some updates so that peerings marked as terminated can be
deleted by users.
2022-08-26 10:52:47 -06:00
Chris S. Kim 937a8ec742 Fix casing 2022-08-26 11:56:26 -04:00
Chris S. Kim 87962b9713 Merge branch 'main' into catalog-service-list-filter 2022-08-26 11:16:06 -04:00
Chris S. Kim e2fe8b8d65 Fix tests for enterprise 2022-08-26 11:14:02 -04:00
Chris S. Kim 1c43a1a7b4 Merge branch 'main' into NET-638-push-server-address-updates-to-the-peer
# Conflicts:
#	agent/grpc-external/services/peerstream/stream_test.go
2022-08-26 10:43:56 -04:00
Chris S. Kim 6ddcc04613
Replace ring buffer with async version (#14314)
We need to watch for changes to peerings and update the server addresses which get served by the ring buffer.

Also, if there is an active connection for a peer, we are getting up-to-date server addresses from the replication stream and can safely ignore the token's addresses which may be stale.
2022-08-26 10:27:13 -04:00
alex 30ff2e9a35
peering: add peer health metric (#14004)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-08-25 16:32:59 -07:00
Chris S. Kim 181063cd23 Exit loop when context is cancelled 2022-08-25 11:48:25 -04:00
cskh 41aea65214
Fix: the inboundconnection limit filter should be placed in front of http co… (#14325)
* fix: the inboundconnection limit should be placed in front of http connection manager

Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2022-08-24 14:13:10 -04:00
Chris S. Kim 8c94d1a80c Update test comment 2022-08-24 13:50:24 -04:00
Chris S. Kim 5f2959329f Add check for zero-length server addresses 2022-08-24 13:30:52 -04:00
skpratt 919da33331
no-op: refactor usagemetrics tests for clarity and DRY cases (#14313) 2022-08-24 12:00:09 -05:00
Pablo Ruiz García 1f293e5244
Added new auto_encrypt.grpc_server_tls config option to control AutoTLS enabling of GRPC Server's TLS usage
Fix for #14253

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
2022-08-24 12:31:38 -04:00
Dan Upton 3b993f2da7
dataplane: update envoy bootstrap params for consul-dataplane (#14017)
Contains 2 changes to the GetEnvoyBootstrapParams response to support
consul-dataplane.

Exposing node_name and node_id:

consul-dataplane will support providing either the node_id or node_name in its
configuration. Unfortunately, supporting both in the xDS meta adds a fair amount
of complexity (partly because most tables are currently indexed on node_name)
so for now we're going to return them both from the bootstrap params endpoint,
allowing consul-dataplane to exchange a node_id for a node_name (which it will
supply in the xDS meta).

Properly setting service for gateways:

To avoid the need to special case gateways in consul-dataplane, service will now
either be the destination service name for connect proxies, or the gateway
service name. This means it can be used as-is in Envoy configuration (i.e. as a
cluster name or in metric tags).
2022-08-24 12:03:15 +01:00
Daniel Upton 13c04a13af proxycfg: terminate stream on irrecoverable errors
This is the OSS portion of enterprise PR 2339.

It improves our handling of "irrecoverable" errors in proxycfg data sources.

The canonical example of this is what happens when the ACL token presented by
Envoy is deleted/revoked. Previously, the stream would get "stuck" until the
xDS server re-checked the token (after 5 minutes) and terminated the stream.

Materializers would also sit burning resources retrying something that could
never succeed.

Now, it is possible for data sources to mark errors as "terminal" which causes
the xDS stream to be closed immediately. Similarly, the submatview.Store will
evict materializers when it observes they have encountered such an error.
2022-08-23 20:17:49 +01:00
Chris S. Kim 81e965479b PR feedback to specify Node name in test mock 2022-08-23 11:51:04 -04:00
Eric Haberkorn 58901ad7df
Cluster peering failover disco chain changes (#14296) 2022-08-23 09:13:43 -04:00
Chris S. Kim cdc8b0634d Fix flakes 2022-08-22 14:45:31 -04:00
Chris S. Kim 03e92826aa Increase heartbeat rate to reduce test flakes 2022-08-22 14:24:05 -04:00
Chris S. Kim 06ba9775ee Remove check for ResponseNonce 2022-08-22 13:55:01 -04:00
Chris S. Kim 547fb9570e Add missing mock assertions 2022-08-22 13:55:01 -04:00
Chris S. Kim adff2eef16 Fix data race
newMockSnapshotHandler has an assertion on t.Cleanup which gets called before the event publisher is cancelled. This commit reorders the context.WithCancel so it properly gets cancelled before the assertion is made.
2022-08-22 13:55:01 -04:00
cskh 060531a29a
Fix: add missing ent meta for test (#14289) 2022-08-22 13:51:04 -04:00
Chris S. Kim 4e40e1d222 Handle server addresses update as client 2022-08-22 13:42:12 -04:00
Chris S. Kim 584d3409c4 Send server addresses on update from server 2022-08-22 13:41:44 -04:00
Chris S. Kim c9d8ad3939 Add new subscription for server addresses 2022-08-22 13:40:25 -04:00
Chris S. Kim 028b87d51f Cleanup unused logger 2022-08-22 13:40:23 -04:00
Chris S. Kim df951bd601 Expose external gRPC port in autopilot
The grpc_port was added to a NodeService's meta in ea58f235f5
2022-08-22 10:07:00 -04:00
cskh 527ebd068a
fix: missing MaxInboundConnections field in service-defaults config entry (#14072)
* fix:  missing max_inbound_connections field in merge config
2022-08-19 14:11:21 -04:00
cskh e84e4b8868
Fix: upgrade pkg imdario/merg to prevent merge config panic (#14237)
* upgrade imdario/merg to prevent merge config panic

* test: service definition takes precedence over service-defaults in merged results
2022-08-17 21:14:04 -04:00
James Hartig f92883bbce Use the maximum jitter when calculating the timeout
The timeout should include the maximum possible
jitter since the server will randomly add to it's
timeout a jitter. If the server's timeout is less
than the client's timeout then the client will
return an i/o deadline reached error.

Before:
```
time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644'
rpc error making call: i/o deadline reached
real    10m11.469s
user    0m0.018s
sys     0m0.023s
```

After:
```
time curl 'http://localhost:8500/v1/catalog/service/service?dc=other-dc&stale=&wait=600s&index=15820644'
[...]
real    10m35.835s
user    0m0.021s
sys     0m0.021s
```
2022-08-17 10:24:09 -04:00
Eric Haberkorn 1a73b0ca20
Add `Targets` field to service resolver failovers. (#14162)
This field will be used for cluster peering failover.
2022-08-15 09:20:25 -04:00
Alessandro De Blasis 5dee555888 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service
Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2022-08-15 08:26:55 +01:00
Alessandro De Blasis ab611eabc3 Merge remote-tracking branch 'hashicorp/main' into feature/health-checks_windows_service
Signed-off-by: Alessandro De Blasis <alex@deblasis.net>
2022-08-15 08:09:56 +01:00
cskh d46b515b64
fix: missing segment and partition (#14194) 2022-08-12 15:21:39 -04:00
Eric Haberkorn ebd5513d4b
Refactor failover code to use Envoy's aggregate clusters (#14178) 2022-08-12 14:30:46 -04:00
cskh 81931e52c3
feat(telemetry): add labels to serf and memberlist metrics (#14161)
* feat(telemetry): add labels to serf and memberlist metrics
* changelog
* doc update

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-08-11 22:09:56 -04:00
Chris S. Kim 4c928cb2f7
Handle breaking change for ServiceVirtualIP restore (#14149)
Consul 1.13.0 changed ServiceVirtualIP to use PeeredServiceName instead of ServiceName which was a breaking change for those using service mesh and wanted to restore their snapshot after upgrading to 1.13.0.

This commit handles existing data with older ServiceName and converts it during restore so that there are no issues when restoring from older snapshots.
2022-08-11 14:47:10 -04:00
Chris S. Kim 3926009405 Add test to verify forwarding 2022-08-11 11:16:02 -04:00
Chris S. Kim 1ef22360c3 Register peerStreamServer internally to enable RPC forwarding 2022-08-11 11:16:02 -04:00
Chris S. Kim de73171202 Handle wrapped errors in isFailedPreconditionErr 2022-08-11 11:16:02 -04:00
Daniel Kimsey 3c4fa9b468 Add support for filtering the 'List Services' API
1. Create a bexpr filter for performing the filtering
2. Change the state store functions to return the raw (not aggregated)
   list of ServiceNodes.
3. Move the aggregate service tags by name logic out of the state store
   functions into a new function called from the RPC endpoint
4. Perform the filtering in the endpoint before aggregation.
2022-08-10 16:52:32 -05:00
cskh 11e7a0d547
fix: shadowed err in retryJoin() (#14112)
- err value will be used later to surface the error message
  if r.join() returns any err.
2022-08-10 10:53:57 -04:00
skpratt 79c23a7cd2
Merge pull request #14056 from hashicorp/proxy-register-port-race
Refactor sidecar_service method to separate port assignment
2022-08-10 09:46:29 -05:00
skpratt aa77559819 Merge branch 'main' into proxy-register-port-race 2022-08-10 08:40:45 -05:00
Chris S. Kim e3046120b3 Close active listeners on error
If startListeners successfully created listeners for some of its input addresses but eventually failed, the function would return an error and existing listeners would not be cleaned up.
2022-08-09 12:22:39 -04:00
Chris S. Kim 6311c651de Add retry in TestAgentConnectCALeafCert_good 2022-08-09 11:20:37 -04:00
Kyle Havlovitz 6938b8c755
Merge pull request #13958 from hashicorp/gateway-wildcard-fix
Fix wildcard picking up services it shouldn't for ingress/terminating gateways
2022-08-08 12:54:40 -07:00
Kyle Havlovitz fe1fcea34f Add some extra handling for destination deletes 2022-08-08 11:38:13 -07:00
freddygv d421e18172 Update snapshot test 2022-08-08 09:17:15 -06:00
freddygv 1031ffc3c7 Re-validate existing secrets at state store
Previously establishment and pending secrets were only checked at the
RPC layer. However, given that these are Check-and-Set transactions we
should ensure that the given secrets are still valid when persisting a
secret exchange or promotion.

Otherwise it would be possible for concurrent requests to overwrite each
other.
2022-08-08 09:06:07 -06:00
freddygv 0ea4bfae94 Test fixes 2022-08-08 08:31:47 -06:00
freddygv c04515a844 Use proto message for each secrets write op
Previously there was a field indicating the operation that triggered a
secrets write. Now there is a message for each operation and it contains
the secret ID being persisted.
2022-08-08 01:41:00 -06:00
Kyle Havlovitz 6580566c3b Update ingress/terminating wildcard logic and handle destinations 2022-08-05 07:56:10 -07:00
freddygv 8067890787 Inherit active secret when exchanging 2022-08-03 17:32:53 -05:00
freddygv 60d6e28c97 Pass explicit signal with op for secrets write
Previously the updates to the peering secrets UUID table relied on
inferring what action triggered the update based on a reconciliation
against the existing secrets.

Instead we now explicitly require the operation to be given so that the
inference isn't necessary. This makes the UUID table logic easier to
reason about and fixes some related bugs.

There is also an update so that the peering secrets get handled on
snapshots/restores.
2022-08-03 17:25:12 -05:00
freddygv 9ca687bc7c Avoid deleting peering secret UUIDs at dialers
Dialers do not keep track of peering secret UUIDs, so they should not
attempt to clean up data from that table when their peering is deleted.

We also now keep peer server addresses when marking peerings for
deletion. Peer server addresses are used by the ShouldDial() helper
when determining whether the peering is for a dialer or an acceptor.
We need to keep this data so that peering secrets can be cleaned up
accordingly.
2022-08-03 16:34:57 -05:00
skpratt 58eed6b049
Merge pull request #13906 from skpratt/validate-port-agent-split
Separate port and socket path validation for local agent
2022-08-02 16:58:41 -05:00
Dhia Ayachi 7154367892
add token to the request when creating a cacheIntentions query (#14005) 2022-08-02 14:27:34 -04:00
Kyle Havlovitz 499211f907 Fix wildcard picking up services it shouldn't for ingress/terminating gateways 2022-08-02 09:41:31 -07:00
Daniel Upton 6452118c15 proxycfg-sources: fix hot loop when service not found in catalog
Fixes a bug where a service getting deleted from the catalog would cause
the ConfigSource to spin in a hot loop attempting to look up the service.

This is because we were returning a nil WatchSet which would always
unblock the select.

Kudos to @freddygv for discovering this!
2022-08-02 15:42:29 +01:00
Freddy 42996411cc
Various peering fixes (#13979)
* Avoid logging StreamSecretID
* Wrap additional errors in stream handler
* Fix flakiness in leader test and rename servers for clarity. There was
  a race condition where the peering was being deleted in the test
  before the stream was active. Now the test waits for the stream to be
  connected on both sides before deleting the associated peering.
* Run flaky test serially
2022-08-01 15:06:18 -06:00
DanStough 169ff71132 fix: ipv4 destination dns resolution 2022-08-01 16:45:57 -04:00
Luke Kysow 988e1fd35d
peering: default to false (#13963)
* defaulting to false because peering will be released as beta
* Ignore peering disabled error in bundles cachetype

Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
Co-authored-by: freddygv <freddy@hashicorp.com>
Co-authored-by: Matt Keeler <mjkeeler7@gmail.com>
2022-08-01 15:22:36 -04:00
Freddy dacf703d20
Merge branch 'main' into fix-kv_entries-metric 2022-08-01 13:19:27 -06:00
Freddy 72b6d69652
Merge pull request #13499 from maxb/delete-unused-metric
Delete definition of metric `consul.acl.blocked.node.deregistration`
2022-08-01 12:31:05 -06:00
Dhia Ayachi 6fd65a4a45
Tgtwy egress HTTP support (#13953)
* add golden files

* add support to http in tgateway egress destination

* fix slice sorting to include both address and port when using server_names

* fix listener loop for http destination

* fix routes to generate a route per port and a virtualhost per port-address combination

* sort virtual hosts list to have a stable order

* extract redundant serviceNode
2022-08-01 14:12:43 -04:00
Matt Keeler f74d0cef7a
Implement/Utilize secrets for Peering Replication Stream (#13977) 2022-08-01 10:33:18 -04:00
alex a45bb1f06b
block PeerName register requests (#13887)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-29 14:36:22 -07:00
Luke Kysow 95096e2c03
peering: retry establishing connection more quickly on certain errors (#13938)
When we receive a FailedPrecondition error, retry that more quickly
because we expect it will resolve shortly. This is particularly
important in the context of Consul servers behind a load balancer
because when establishing a connection we have to retry until we
randomly land on a leader node.

The default retry backoff goes from 2s, 4s, 8s, etc. which can result in
very long delays quite quickly. Instead, this backoff retries in 8ms
five times, then goes exponentially from there: 16ms, 32ms, ... up to a
max of 8152ms.
2022-07-29 13:04:32 -07:00
Sarah Pratt 10a4999a87 Separate port and socket path requirement in case of local agent assignment 2022-07-29 13:28:21 -05:00
alex 92c615c35f
Merge pull request #13952 from hashicorp/sync-more-acl
sync more acl enforcement
2022-07-28 12:31:02 -07:00
Dhia Ayachi 256694b603
inject gateway addons to destination clusters (#13951) 2022-07-28 15:17:35 -04:00
acpana eae4e71492
sync more acl enforcement
sync w ent at 32756f7

Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-28 12:01:52 -07:00
alex 41f3343eac
Merge pull request #13929 from hashicorp/fix-validation
[sync] fix empty partitions matching
2022-07-28 10:14:49 -07:00
Sarah Pratt a3ef6f016e refactor sidecare_service method into parts 2022-07-28 09:07:13 -05:00
Ashwin Venkatesh eef9edaed9
Add peer counts to emitted metrics. (#13930) 2022-07-27 18:34:04 -04:00
Luke Kysow 465a9801e1
Merge pull request #13924 from hashicorp/lkysow/util-metric-peering
peering: don't track imported services/nodes in usage
2022-07-27 14:49:55 -07:00
acpana 6033584349
use EqualPartitions
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-27 14:48:30 -07:00
acpana 0351ca5136
better fix
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-27 14:28:08 -07:00
acpana 8b2ef80336
sync w ent
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-27 11:41:39 -07:00
Chris S. Kim 0999e05a7d Reduce arm64 flakes for TestConnectCA_ConfigurationSet_ChangeKeyConfig_Primary
There were 16 combinations of tests but 4 of them were duplicates since the default key type and bits were "ec" and 256. That entry was commented out to reduce the subtest count to 12.

testrpc.WaitForLeader was failing on arm64 environments; the cause is unknown but it might be due to the environment being flooded with parallel tests making RPC calls. The RPC polling+retry was replaced with a simpler check for leadership based on raft.
2022-07-27 13:54:34 -04:00
Chris S. Kim 8ead1caf53 Retry checks for virtual IP metadata 2022-07-27 13:54:34 -04:00
Chris S. Kim 62ed0250c3 Sort slice of ServiceNames deterministically 2022-07-27 13:54:34 -04:00
Sarah Pratt f520f6dd0f Separate port and socket path requirement in case of local agent assignment 2022-07-27 12:30:52 -05:00
Luke Kysow 740d54e730 peering: don't track imported services/nodes in usage
Services/nodes that are imported from other peers are stored in
state. We don't want to count those as part of our own cluster's usage.
2022-07-27 09:08:51 -07:00
cskh 4e292b7b72
chore: clarify the error message: service.service must not be empty (#13907)
- when register service using catalog endpoint, the key of service
  name actually should be "service". Add this information to the
  error message will help user to quickly fix in the request.
2022-07-27 10:16:46 -04:00
cskh 59e81a728e
chore: removed unused method AddService (#13905)
- This AddService is not used anywhere.
  AddServiceWithChecks is place of AddService
- Test code is updated
2022-07-26 16:54:53 -04:00
Luke Kysow 021b00e321 Remove duplicate comment 2022-07-26 10:19:49 -07:00
alex 437a28d18a
peering: prevent peering in same partition (#13851)
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-07-25 18:00:48 -07:00
Nitya Dhanushkodi 27bd895ac8
peering: remove validation that forces peering token server addresses to be an IP, allow hostname based addresses (#13874) 2022-07-25 16:33:47 -07:00
Luke Kysow 8c5b70d227
Rename receive to recv in tracker (#13896)
Because it's shorter
2022-07-25 16:08:03 -07:00
Luke Kysow 3530d3782d
peering: read endpoints can now return failing status (#13849)
Track streams that have been disconnected due to an error and
set their statuses to failing.
2022-07-25 14:27:53 -07:00
Kyle Havlovitz 93de25f87c
Merge pull request #13872 from hashicorp/remove-upstream-log
Remove extra logging from ingress upstream watch shutdown
2022-07-25 12:55:30 -07:00
Chris S. Kim 73a84f256f
Preserve PeeringState on upsert (#13666)
Fixes a bug where if the generate token is called twice, the second call upserts the zero-value (undefined) of PeeringState.
2022-07-25 14:37:56 -04:00
Chris S. Kim 8ed49ea4d0
Update envoy metrics label extraction for peered clusters and listeners (#13818)
Now that peered upstreams can generate envoy resources (#13758), we need a way to disambiguate local from peered resources in our metrics. The key difference is that datacenter and partition will be replaced with peer, since in the context of peered resources partition is ambiguous (could refer to the partition in a remote cluster or one that exists locally). The partition and datacenter of the proxy will always be that of the source service.

Regexes were updated to make emitting datacenter and partition labels mutually exclusive with peer labels.

Listener filter names were updated to better match the existing regex.

Cluster names assigned to peered upstreams were updated to be synthesized from local peer name (it previously used the externally provided primary SNI, which contained the peer name from the other side of the peering). Integration tests were updated to assert for the new peer labels.
2022-07-25 13:49:00 -04:00
DanStough 2da8949d78 feat: convert destination address to slice 2022-07-25 12:31:58 -04:00
Freddy f03cca7576
[OSS] Add ACL enforcement to peering endpoints (#13878) 2022-07-25 10:04:10 -06:00
Matt Keeler 58e4d8235b
Enable/Disable Peering Support in the UI (#13816)
We enabled/disable based on the config flag.
2022-07-25 11:50:11 -04:00
freddygv b544ce6485 Add ACL enforcement to peering endpoints 2022-07-25 09:34:29 -06:00
Kyle Havlovitz 016f963e7e Remove excess debug log from ingress upstream shutdown 2022-07-22 17:29:38 -07:00
alex 279d458e6e
peering: use ShouldDial to validate peer role (#13823)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-22 15:56:25 -07:00
Luke Kysow a1e6d69454
peering: add config to enable/disable peering (#13867)
* peering: add config to enable/disable peering

Add config:

```
peering {
  enabled = true
}
```

Defaults to true. When disabled:
1. All peering RPC endpoints will return an error
2. Leader won't start its peering establishment goroutines
3. Leader won't start its peering deletion goroutines
2022-07-22 15:20:21 -07:00
Kyle Havlovitz 0786517b56
Merge pull request #13847 from hashicorp/gateway-goroutine-leak
Fix goroutine leaks in proxycfg when using ingress gateway
2022-07-22 14:43:22 -07:00
Freddy f99df57840
[OSS] Add new peering ACL rule (#13848)
This commit adds a new ACL rule named "peering" to authorize
actions taken against peering-related endpoints.

The "peering" rule has several key properties:
- It is scoped to a partition, and MUST be defined in the default
  namespace.

- Its access level must be "read', "write", or "deny".

- Granting an access level will apply to all peerings. This ACL rule
  cannot be used to selective grant access to some peerings but not
  others.

- If the peering rule is not specified, we fall back to the "operator"
  rule and then the default ACL rule.
2022-07-22 14:42:23 -06:00
alex 927cee692b
peering: emit exported services count metric (#13811)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-22 12:05:08 -07:00
Daniel Upton a8df87f574 proxycfg-glue: server-local implementation of `ExportedPeeredServices`
This is the OSS portion of enterprise PR 2377.

Adds a server-local implementation of the proxycfg.ExportedPeeredServices
interface that sources data from a blocking query against the server's
state store.
2022-07-22 15:23:23 +01:00
Eric Haberkorn 501089292e
Add Cluster Peering Failover Support to Prepared Queries (#13835)
Add peering failover support to prepared queries
2022-07-22 09:14:43 -04:00
Nitya Dhanushkodi f47319b7c6
update generate token endpoint to take external addresses (#13844)
Update generate token endpoint (rpc, http, and api module)

If ServerExternalAddresses are set, it will override any addresses gotten from the "consul" service, and be used in the token instead, and dialed by the dialer. This allows for setting up a load balancer for example, in front of the consul servers.
2022-07-21 14:56:11 -07:00
acpana 12b773ab02
Rename peering internal to ~
sync ENT to 5679392c81

Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-21 10:51:05 -07:00
Luke Kysow 0c87be0845
peering: Add heartbeating to peering streams (#13806)
* Add heartbeating to peering streams
2022-07-21 10:03:27 -07:00
Daniel Upton 3655802fdc proxycfg-glue: server-local implementation of `PeeredUpstreams`
This is the OSS portion of enterprise PR 2352.

It adds a server-local implementation of the proxycfg.PeeredUpstreams interface
based on a blocking query against the server's state store.

It also fixes an omission in the Virtual IP freeing logic where we were never
updating the max index (and therefore blocking queries against
VirtualIPsForAllImportedServices would not return on service deletion).
2022-07-21 13:51:59 +01:00
Luke Kysow c411e6b326
Add send mutex to protect against concurrent sends (#13805) 2022-07-20 15:48:18 -07:00
Kyle Havlovitz 0be7d923dc Cancel upstream watches when the discovery chain has been removed 2022-07-20 14:26:52 -07:00
Kyle Havlovitz 31318d7049 Fix duplicate Notify calls for discovery chains in ingress gateways 2022-07-20 14:25:20 -07:00
Evan Culver 4116537b83
connect: Add support for Envoy 1.23, remove 1.19 (#13807) 2022-07-19 14:51:04 -07:00
Paul Glass 77afe0e76e
Extract AWS auth implementation out of Consul (#13760) 2022-07-19 16:26:44 -05:00
Chris S. Kim 495936300e
Make envoy resources for inferred peered upstreams (#13758)
Peered upstreams has a separate loop in xds from discovery chain upstreams. This PR adds similar but slightly modified code to add filters for peered upstream listeners, clusters, and endpoints in the case of transparent proxy.
2022-07-19 14:56:28 -04:00
alex de5a991d8c
peering: refactor reconcile, cleanup (#13795)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-19 11:43:29 -07:00
Luke Kysow e8d965e56f
peerstream: set keepalive enforcement to 15s (#13796)
The client is set to send keepalive pings every 30s. The server
keepalive enforcement must be set to a number less than that,
otherwise it will disconnect clients for sending pings too often.
MinTime governs the minimum amount of time between pings.
2022-07-18 16:12:03 -07:00
alex a9ae2ff4fa
peering: track exported services (#13784)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-07-18 10:20:04 -07:00
R.B. Boyer cd513aeead
peerstream: require a resource subscription to receive updates of that type (#13767)
This mimics xDS's discovery protocol where you must request a resource
explicitly for the exporting side to send those events to you.

As part of this I aligned the overall ResourceURL with the TypeURL that
gets embedded into the encoded protobuf Any construct. The
CheckServiceNodes is now wrapped in a better named "ExportedService"
struct now.
2022-07-15 15:03:40 -05:00
R.B. Boyer c737301093
peerstream: fix test assertions (#13780) 2022-07-15 14:43:24 -05:00
Luke Kysow 46381b1a7f
Add docs for peerStreamServer vs peeringServer. (#13781) 2022-07-15 12:23:05 -07:00
Luke Kysow ca3d7c964c
peerstream: dialer should reconnect when stream closes (#13745)
* peerstream: dialer should reconnect when stream closes

If the stream is closed unexpectedly (i.e. when we haven't received
a terminated message), the dialer should attempt to re-establish the
stream.

Previously, the `HandleStream` would return `nil` when the stream
was closed. The caller then assumed the stream was terminated on purpose
and so didn't reconnect when instead it was stopped unexpectedly and
the dialer should have attempted to reconnect.
2022-07-15 11:58:33 -07:00
R.B. Boyer bb4d4040fb
server: ensure peer replication can successfully use TLS over external gRPC (#13733)
Ensure that the peer stream replication rpc can successfully be used with TLS activated.

Also:

- If key material is configured for the gRPC port but HTTPS is not
  enabled now TLS will still be activated for the gRPC port.

- peerstream replication stream opened by the establishing-side will now
  ignore grpc.WithBlock so that TLS errors will bubble up instead of
  being awkwardly delayed or suppressed
2022-07-15 13:15:50 -05:00
alex adb5ffa1a6
peering: track imported services (#13718) 2022-07-15 10:20:43 -07:00
Matt Keeler 257f88d4df
Use Node Name for peering healthSnapshot instead of ID (#13773)
A Node ID is not a required field with Consul’s data model. Therefore we cannot reliably expect all uses to have it. However the node name is required and must be unique so its equally as good of a key for the internal healthSnapshot node tracking.
2022-07-15 10:51:38 -04:00
Matt Keeler 05b5e7e2ca
Enable partition support for peering establishment (#13772)
Prior to this the dialing side of the peering would only ever work within the default partition. This commit allows properly parsing the partition field out of the API struct request body, query param and header.
2022-07-15 10:07:07 -04:00
Dan Stough 49f3dadb8f feat: connect proxy xDS for destinations
Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>
2022-07-14 15:27:02 -04:00
Daniel Upton 3d74efa8ad proxycfg-glue: server-local implementation of `FederationStateListMeshGateways`
This is the OSS portion of enterprise PR 2265.

This PR provides a server-local implementation of the
proxycfg.FederationStateListMeshGateways interface based on blocking queries.
2022-07-14 18:22:12 +01:00
Daniel Upton ccc672013e proxycfg-glue: server-local implementation of `GatewayServices`
This is the OSS portion of enterprise PR 2259.

This PR provides a server-local implementation of the proxycfg.GatewayServices
interface based on blocking queries.
2022-07-14 18:22:12 +01:00
Daniel Upton 15a319dbfe proxycfg-glue: server-local implementation of `TrustBundle` and `TrustBundleList`
This is the OSS portion of enterprise PR 2250.

This PR provides server-local implementations of the proxycfg.TrustBundle and
proxycfg.TrustBundleList interfaces, based on local blocking queries.
2022-07-14 18:22:12 +01:00
Daniel Upton 673d02d30f proxycfg-glue: server-local implementation of the `Health` interface
This is the OSS portion of enterprise PR 2249.

This PR introduces an implementation of the proxycfg.Health interface based on a
local materialized view of the health events.

It reuses the view and request machinery from agent/rpcclient/health, which made
it super straightforward.
2022-07-14 18:22:12 +01:00
Daniel Upton 3c533ceea8 proxycfg-glue: server-local implementation of `ServiceList`
This is the OSS portion of enterprise PR 2242.

This PR introduces a server-local implementation of the proxycfg.ServiceList
interface, backed by streaming events and a local materializer.
2022-07-14 18:22:12 +01:00
Daniel Upton fbf88d3b19 proxycfg-glue: server-local compiled discovery chain data source
This is the OSS portion of enterprise PR 2236.

Adds a local blocking query-based implementation of the proxycfg.CompiledDiscoveryChain interface.
2022-07-14 18:22:12 +01:00
Chris S. Kim f56810132f Check if an upstream is implicit from either intentions or peered services 2022-07-13 16:53:20 -04:00
Chris S. Kim 02cff2394d Use new maps for proxycfg peered data 2022-07-13 16:05:10 -04:00
Chris S. Kim 7f32cba735 Add new watch.Map type to refactor proxycfg 2022-07-13 16:05:10 -04:00
Chris S. Kim b4ffa9ae0c Scrub VirtualIPs before exporting 2022-07-13 16:05:10 -04:00
Kyle Havlovitz 9097e2b0f0
Merge pull request #13699 from hashicorp/tgate-http2-upstream
Respect http2 protocol for upstreams of terminating gateways
2022-07-13 09:41:15 -07:00
Dan Upton b9e525d689
grpc: rename public/private directories to external/internal (#13721)
Previously, public referred to gRPC services that are both exposed on
the dedicated gRPC port and have their definitions in the proto-public
directory (so were considered usable by 3rd parties). Whereas private
referred to services on the multiplexed server port that are only usable
by agents and other servers.

Now, we're splitting these definitions, such that external/internal
refers to the port and public/private refers to whether they can be used
by 3rd parties.

This is necessary because the peering replication API needs to be
exposed on the dedicated port, but is not (yet) suitable for use by 3rd
parties.
2022-07-13 16:33:48 +01:00
R.B. Boyer 30fffd0c90
peerstream: some cosmetic refactors to make this easier to follow (#13732)
- Use some protobuf construction helper methods for brevity.
- Rename a local variable to avoid later shadowing.
- Rename the Nonce field to be more like xDS's naming.
- Be more explicit about which PeerID fields are empty.
2022-07-13 10:00:35 -05:00
Kyle Havlovitz 7d0c692374 Use protocol from resolved config entry, not gateway service 2022-07-12 16:23:40 -07:00
Kyle Havlovitz 7162e3bde2 Enable http2 options for grpc protocol 2022-07-12 14:38:44 -07:00
R.B. Boyer c5c216008d
peering: always send the mesh gateway SpiffeID even for tcp services (#13728)
If someone were to switch a peer-exported service from L4 to L7 there
would be a brief SAN validation hiccup as traffic shifted to the mesh
gateway for termination.

This PR sends the mesh gateway SpiffeID down all the time so the clients
always expect a switch.
2022-07-12 11:38:13 -05:00
R.B. Boyer f0e6e4e697
state: prohibit changing an exported tcp discovery chain in a way that would break SAN validation (#13727)
For L4/tcp exported services the mesh gateways will not be terminating
TLS. A caller in one peer will be directly establishing TLS connections
to the ultimate exported service in the other peer.

The caller will be doing SAN validation using the replicated SpiffeID
values shipped from the exporting side. There are a class of discovery
chain edits that could be done on the exporting side that would cause
the introduction of a new SpiffeID value. In between the time of the
config entry update on the exporting side and the importing side getting
updated peer stream data requests to the exported service would fail due
to SAN validation errors.

This is unacceptable so instead prohibit the exporting peer from making
changes that would break peering in this way.
2022-07-12 11:17:33 -05:00
R.B. Boyer 2317f37b4d
state: prohibit exported discovery chains to have cross-datacenter or cross-partition references (#13726)
Because peerings are pairwise, between two tuples of (datacenter,
partition) having any exported reference via a discovery chain that
crosses out of the peered datacenter or partition will ultimately not be
able to work for various reasons. The biggest one is that there is no
way in the ultimate destination to configure an intention that can allow
an external SpiffeID to access a service.

This PR ensures that a user simply cannot do this, so they won't run
into weird situations like this.
2022-07-12 11:03:41 -05:00
Chris S. Kim a6634db4a5
Return error if ServerAddresses is empty (#13714) 2022-07-12 11:09:00 -04:00
Kyle Havlovitz 439eccdd80 Respect http2 protocol for upstreams of terminating gateways 2022-07-08 14:30:45 -07:00
R.B. Boyer af04851637
peering: move peer replication to the external gRPC port (#13698)
Peer replication is intended to be between separate Consul installs and
effectively should be considered "external". This PR moves the peer
stream replication bidirectional RPC endpoint to the external gRPC
server and ensures that things continue to function.
2022-07-08 12:01:13 -05:00
R.B. Boyer ea58f235f5
server: broadcast the public grpc port using lan serf and update the consul service in the catalog with the same data (#13687)
Currently servers exchange information about their WAN serf port
and RPC port with serf tags, so that they all learn of each other's
addressing information. We intend to make larger use of the new
public-facing gRPC port exposed on all of the servers, so this PR
addresses that by passing around the gRPC port via serf tags and
then ensuring the generated consul service in the catalog has
metadata about that new port as well for ease of non-serf-based lookup.
2022-07-07 13:55:41 -05:00
Freddy 3542138e4d
Parse peer name for virtual IP DNS queries (#13602)
This commit updates the DNS query locality parsing so that the virtual
IP for an imported service can be queried.

Note that:
- Support for parsing a peer in other service discovery queries was not
  added.
- Querying another datacenter for a virtual IP is not supported. This
  was technically allowed in 1.11 but is being rolled back for 1.13
  because it is not a use-case we intended to support. Virtual IPs in
  different datacenters are going to collide because they are allocated
  sequentially.
2022-07-06 10:30:04 -06:00
R.B. Boyer 2a945facec
test: update mockery use to put mocks into test files (#13656)
--testonly doesn't do anything anymore so switch to --filename instead
2022-07-05 16:57:15 -05:00
Chris S. Kim f07132dacc
Revise possible states for a peering. (#13661)
These changes are primarily for Consul's UI, where we want to be more
specific about the state a peering is in.

- The "initial" state was renamed to pending, and no longer applies to
  peerings being established from a peering token.

- Upon request to establish a peering from a peering token, peerings
  will be set as "establishing". This will help distinguish between the
  two roles: the cluster that generates the peering token and the
  cluster that establishes the peering.

- When marked for deletion, peering state will be set to "deleting".
  This way the UI determines the deletion via the state rather than the
  "DeletedAt" field.

Co-authored-by: freddygv <freddy@hashicorp.com>
2022-07-04 10:47:58 -04:00
Daniel Upton 45886848b4 proxycfg: server-local intention upstreams data source
This is the OSS portion of enterprise PR 2157.

It builds on the local blocking query work in #13438 to implement the
proxycfg.IntentionUpstreams interface using server-local data.

Also moves the ACL filtering logic from agent/consul into the acl/filter
package so that it can be reused here.
2022-07-04 10:48:36 +01:00
Daniel Upton 37ccbd2826 proxycfg: server-local intentions data source
This is the OSS portion of enterprise PR 2141.

This commit provides a server-local implementation of the `proxycfg.Intentions`
interface that sources data from streaming events.

It adds events for the `service-intentions` config entry type, and then consumes
event streams (via materialized views) for the service's explicit intentions and
any applicable wildcard intentions, merging them into a single list of intentions.

An alternative approach I considered was to consume _all_ intention events (via
`SubjectWildcard`) and filter out the irrelevant ones. This would admittedly
remove some complexity in the `agent/proxycfg-glue` package but at the expense
of considerable overhead from waking potentially many thousands of connect
proxies every time any intention is updated.
2022-07-04 10:48:36 +01:00
Daniel Upton 653b8c4f9d proxycfg: server-local config entry data sources
This is the OSS portion of enterprise PR 2056.

This commit provides server-local implementations of the proxycfg.ConfigEntry
and proxycfg.ConfigEntryList interfaces, that source data from streaming events.

It makes use of the LocalMaterializer type introduced for peering replication,
adding the necessary support for authorization.

It also adds support for "wildcard" subscriptions (within a topic) to the event
publisher, as this is needed to fetch service-resolvers for all services when
configuring mesh gateways.

Currently, events will be emitted for just the ingress-gateway, service-resolver,
and mesh config entry types, as these are the only entries required by proxycfg
— the events will be emitted on topics named IngressGateway, ServiceResolver,
and MeshConfig topics respectively.

Though these events will only be consumed "locally" for now, they can also be
consumed via the gRPC endpoint (confirmed using grpcurl) so using them from
client agents should be a case of swapping the LocalMaterializer for an
RPCMaterializer.
2022-07-04 10:48:36 +01:00
alex cd9ca4290a
peering: add imported/exported counts to peering (#13644)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-06-29 14:07:30 -07:00
Chris S. Kim b186731a2e
Fix ENT drift in files (#13647) 2022-06-29 16:53:22 -04:00
Chris S. Kim d8b7940e40
Add internal endpoint to fetch peered upstream candidates from VirtualIP table (#13642)
For initial cluster peering TProxy support we consider all imported services of a partition to be potential upstreams.

We leverage the VirtualIP table because it stores plain service names (e.g. "api", not "api-sidecar-proxy").
2022-06-29 16:34:58 -04:00
Eric Haberkorn 653cb42944
Fix spelling mistake in serverless patcher (#13607)
passhthrough -> passthrough
2022-06-29 15:21:21 -04:00
alex 07bc22e405
no 1.9 style metrics (#13532)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-29 09:46:37 -07:00
alex beb8b03e8a
peering: reconcile/ hint active state for list (#13619)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-29 09:43:50 -07:00
R.B. Boyer 31b95c747b
xds: modify rbac rules to use the XFCC header for peered L7 enforcement (#13629)
When the protocol is http-like, and an intention has a peered source
then the normal RBAC mTLS SAN field check is replaces with a joint combo
of:

    mTLS SAN field must be the service's local mesh gateway leaf cert
      AND
    the first XFCC header (from the MGW) must have a URI field that matches the original intention source

Also:

- Update the regex program limit to be much higher than the teeny
  defaults, since the RBAC regex constructions are more complicated now.

- Fix a few stray panics in xds generation.
2022-06-29 10:29:54 -05:00
R.B. Boyer de0f9ac519
xds: have mesh gateways forward peered SpiffeIDs using the XFCC header (#13625) 2022-06-28 15:32:42 -05:00
R.B. Boyer 1a9c86ea8f
xds: mesh gateways now correctly load up peer-exported discovery chains using L7 protocols (#13624)
A mesh gateway will now configure the filter chains for L7 exported
services using the correct discovery chain information.
2022-06-28 14:52:25 -05:00
R.B. Boyer 0fa828db76
peering: replicate all SpiffeID values necessary for the importing side to do SAN validation (#13612)
When traversing an exported peered service, the discovery chain
evaluation at the other side may re-route the request to a variety of
endpoints. Furthermore we intend to terminate mTLS at the mesh gateway
for arriving peered traffic that is http-like (L7), so the caller needs
to know the mesh gateway's SpiffeID in that case as well.

The following new SpiffeID values will be shipped back in the peerstream
replication:

- tcp: all possible SpiffeIDs resulting from the service-resolver
        component of the exported discovery chain

- http-like: the SpiffeID of the mesh gateway
2022-06-27 14:37:18 -05:00
Max Bowsher ef4b9e541f
Merge branch 'main' into fix-kv_entries-metric 2022-06-27 18:57:03 +01:00
alex 53f0cf5835
peering, internal: support UIServices, UINodes, UINodeInfo (#13577) 2022-06-24 15:17:35 -07:00
Chris S. Kim 2e4cb6f77d
Add new index for PeeredServiceName and ServiceVirtualIP (#13582)
For TProxy we will be leveraging the VirtualIP table, which needs to become peer-aware
2022-06-24 14:38:39 -04:00
alex 20ecf0febd
Merge pull request #13570 from hashicorp/acpance/peering-oss-intentions
oss: peering, http: get peer service intentions (#2098)
2022-06-23 08:15:59 -07:00
Will Jordan 34ecbc1d71
Add per-node max indexes (#12399)
Adds fine-grained node.[node] entries to the index table, allowing blocking queries to return fine-grained indexes that prevent them from returning immediately when unrelated nodes/services are updated.

Co-authored-by: kisunji <ckim@hashicorp.com>
2022-06-23 11:13:25 -04:00
Chris S. Kim ba89a7d9b0
Make memdb indexers generic (#13558)
We have many indexer functions in Consul which take interface{} and type assert before building the index. We can use generics to get rid of the initial plumbing and pass around functions with better defined signatures. This has two benefits: 1) Less verbosity; 2) Developers can parse the argument types to memdb schemas without having to introspect the function for the type assertion.
2022-06-23 11:07:19 -04:00
Matt Keeler 7a4d13b0b2
Port over the index 0 -> 1 code that lived in the old rpc setQueryMeta function. (#13561) 2022-06-23 09:34:47 -04:00
acpana 99c2e11328
oss: peering, http: get peer service intentions (#2098)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-22 16:25:09 -07:00
R.B. Boyer e8ea3d7c3b
state: peering ID assignment cannot happen inside of the state store (#13525)
Move peering ID assignment outisde of the FSM, so that the ID is written
to the raft log and the same ID is used by all voters, and after
restarts.
2022-06-21 13:04:08 -05:00
Matt Keeler cb01702cd2
Add server local blocking queries and watches (#13438)
Co-authored-by: Dan Upton <daniel@floppy.co>
2022-06-21 13:36:49 -04:00
Chris S. Kim fb5eb20563
Pass trust domain to RBAC to validate and fix use of wrong peer trust bundles (#13508) 2022-06-20 22:47:14 -04:00
Max Bowsher 7b97b8abd2 Delete definition of metric `consul.acl.blocked.node.registration`
Although the metric is defined, there is no code which ever sets its
value - the code in question is genuinely asymmetric - there are 3 types
of object for which registration can be tracked, but only 2 for which
deregistration can be tracked.
2022-06-19 17:38:04 +01:00
Max Bowsher 7c19c701e1 Fix incorrect name and doc for kv_entries metric
The name of the metric as registered with the metrics library to provide
the help string, was incorrect compared with the actual code that sets
the metric value - bring them into sync.

Also, the help message was incorrect. Rather than copy the help message
from telemetry.mdx, which was correct, but felt a bit unnatural in the
way it was worded, update both of them to a new wording.
2022-06-19 11:58:23 +01:00
Dan Upton e00e3a0bc3
Move ACLResolveResult into acl/resolver package (#13467)
Having this type live in the agent/consul package makes it difficult to
put anything that relies on token resolution (e.g. the new gRPC services)
in separate packages without introducing import cycles.

For example, if package foo imports agent/consul for the ACLResolveResult
type it means that agent/consul cannot import foo to register its service.

We've previously worked around this by wrapping the ACLResolver to
"downgrade" its return type to an acl.Authorizer - aside from the
added complexity, this also loses the resolved identity information.

In the future, we may want to move the whole ACLResolver into the
acl/resolver package. For now, putting the result type there at least,
fixes the immediate import cycle issues.
2022-06-17 10:24:43 +01:00
DanStough 4b402e3119 feat: tgtwy xDS generation for destinations
Signed-off-by: Dhia Ayachi <dhia@hashicorp.com>
2022-06-16 16:17:49 -04:00
alex bd4ddb3720
peering: block Intention.Apply ops (#13451)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-16 12:07:28 -07:00
alex b3e99784a6
peering, state: account for peer intentions (#13443)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-16 10:27:31 -07:00
R.B. Boyer da8cea58c9
xds: begin refactor to always pass test snapshots through all xDS types (#13461) 2022-06-15 14:58:28 -05:00
R.B. Boyer 201d1458c3
xds: mesh gateways now have their own leaf certificate when involved in a peering (#13460)
This is only configured in xDS when a service with an L7 protocol is
exported.

They also load any relevant trust bundles for the peered services to
eventually use for L7 SPIFFE validation during mTLS termination.
2022-06-15 14:36:18 -05:00
Riddhi Shah 411edc876b
[OSS] Support merge-central-config option in node services list API (#13450)
Adds the merge-central-config query param option to the /catalog/node-services/:node-name API,
to get a service definition in the response that is merged with central defaults (proxy-defaults/service-defaults).

Updated the consul connect envoy command to use this option when
retrieving the proxy service details so as to render the bootstrap configuration correctly.
2022-06-15 08:30:31 -07:00
Evan Culver 7f8c650d61
connect: Use Envoy 1.22.2 instead of 1.22.1 (#13444) 2022-06-14 15:29:41 -07:00
freddygv f3843809da Avoid deleting peerings marked as terminated.
When our peer deletes the peering it is locally marked as terminated.
This termination should kick off deleting all imported data, but should
not delete the peering object itself.

Keeping peerings marked as terminated acts as a signal that the action
took place.
2022-06-14 15:37:09 -06:00
freddygv 6453375ab2 Add leader routine to clean up peerings
Once a peering is marked for deletion a new leader routine will now
clean up all imported resources and then the peering itself.

A lot of the logic was grabbed from the namespace/partitions deferred
deletions but with a handful of simplifications:
- The rate limiting is not configurable.

- Deleting imported nodes/services/checks is done by deleting nodes with
  the Txn API. The services and checks are deleted as a side-effect.

- There is no "round rate limiter" like with namespaces and partitions.
  This is because peerings are purely local, and deleting a peering in
  the datacenter does not depend on deleting data from other DCs like
  with WAN-federated namespaces. All rate limiting is handled by the
  Raft rate limiter.
2022-06-14 15:36:50 -06:00
Evan Culver ba6136eb42
connect: Update Envoy support matrix to latest patch releases (#13431) 2022-06-14 13:19:09 -07:00
alex a0a49ce2a6
peering: intentions list test (#13435) 2022-06-14 10:59:53 -07:00
freddygv 6c8ab1bbac Fixup stream tear-down steps.
1. Fix a bug where the peering leader routine would not track all active
   peerings in the "stored" reconciliation map. This could lead to
   tearing down streams where the token was generated, since the
   ConnectedStreams() method used for reconciliation returns all streams
   and not just the ones initiated by this leader routine.

2. Fix a race where stream contexts were being canceled before
   termination messages were being processed by a peer.

   Previously the leader routine would tear down streams by canceling
   their context right after the termination message was sent. This
   context cancelation could be propagated to the server side faster
   than the termination message. Now there is a change where the
   dialing peer uses CloseSend() to signal when no more messages will
   be sent. Eventually the server peer will read an EOF after receiving
   and processing the preceding termination message.

   Using CloseSend() is actually not enough to address the issue
   mentioned, since it doesn't wait for the server peer to finish
   processing messages. Because of this now the dialing peer also reads
   from the stream until an error signals that there are no more
   messages. Receiving an EOF from our peer indicates that they
   processed the termination message and have no additional work to do.

   Given that the stream is being closed, all the messages received by
   Recv are discarded. We only check for errors to avoid importing new
   data.
2022-06-13 12:10:42 -06:00
freddygv cc921a9c78 Update peering state and RPC for deferred deletion
When deleting a peering we do not want to delete the peering and all
imported data in a single operation, since deleting a large amount of
data at once could overload Consul.

Instead we defer deletion of peerings so that:

1. When a peering deletion request is received via gRPC the peering is
   marked for deletion by setting the DeletedAt field.

2. A leader routine will monitor for peerings that are marked for
   deletion and kick off a throttled deletion of all imported resources
   before deleting the peering itself.

This commit mostly addresses point #1 by modifying the peering service
to mark peerings for deletion. Another key change is to add a
PeeringListDeleted state store function which can return all peerings
marked for deletion. This function is what will be watched by the
deferred deletion leader routine.
2022-06-13 12:10:32 -06:00
Freddy 71b254522e
Clean up imported nodes/services/checks as needed (#13367)
Previously, imported data would never be deleted. As
nodes/services/checks were registered and deregistered, resources
deleted from the exporting cluster would accumulate in the imported
cluster.

This commit makes updates to replication so that whenever an update is
received for a service name we reconcile what was present in the catalog
against what was received.

This handleUpdateService method can handle both updates and deletions.
2022-06-13 11:52:28 -06:00
Mark Anderson edbf19f4e8
Merge pull request #13357 from hashicorp/ma/add-build-date-oss
Add build date (oss)
2022-06-13 08:43:20 -07:00
Chris S. Kim a02e9abcc1
Update RBAC to handle imported services (#13404)
When converting from Consul intentions to xds RBAC rules, services imported from other peers must encode additional data like partition (from the remote cluster) and trust domain.

This PR updates the PeeringTrustBundle to hold the sending side's local partition as ExportedPartition. It also updates RBAC code to encode SpiffeIDs of imported services with the ExportedPartition and TrustDomain.
2022-06-10 17:15:22 -04:00
R.B. Boyer f557509e58
xds: allow for peered upstreams to use tagged addresses that are hostnames (#13422)
Mesh gateways can use hostnames in their tagged addresses (#7999). This is useful
if you were to expose a mesh gateway using a cloud networking load balancer appliance
that gives you a DNS name but no reliable static IPs.

Envoy cannot accept hostnames via EDS and those must be configured using CDS.
There was already logic when configuring gateways in other locations in the code, but
given the illusions in play for peering the downstream of a peered service wasn't aware
that it should be doing that.

Also:
- ensuring that we always try to use wan-like addresses to cross peer boundaries.
2022-06-10 16:11:40 -05:00
Kyle Havlovitz 7f62571419 Add dns node lookup support in partitions 2022-06-10 11:23:51 -07:00
R.B. Boyer 7001e1151c
peering: rename initiate to establish in the context of the APIs (#13419) 2022-06-10 11:10:46 -05:00
Mark Anderson dd22ceccd1 Change default dates
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-06-09 17:07:41 -07:00
Mark Anderson f65093f1c6 Fixup some more tests
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-06-09 17:04:05 -07:00
Mark Anderson 19c87be3a6 Add build date to self endpoint
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-06-09 17:04:05 -07:00
Mark Anderson ec060e5e37 Build date in config file
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2022-06-09 17:04:05 -07:00
R.B. Boyer bba3eb8cdd
peering: mesh gateways are required for cross-peer service mesh communication (#13410)
Require use of mesh gateways in order for service mesh data plane
traffic to flow between peers.

This also adds plumbing for envoy integration tests involving peers, and
one starter peering test.
2022-06-09 11:05:18 -05:00
Alessandro De Blasis 06304bfb0d lint: conversion 2022-06-09 16:17:20 +01:00
Alessandro De Blasis 28f19e4627 tests: removed redundant probe test 2022-06-09 15:49:45 +01:00
Alessandro De Blasis af083cc5ba tests: added syscall mocking and tests for Check_OSService 2022-06-09 15:48:34 +01:00
kisunji 196a1c468a Add missing index for read 2022-06-08 13:53:31 -04:00
kisunji d026d84880 Add IntentionMatch tests for source peers 2022-06-08 13:53:31 -04:00
kisunji bb0b42da12 Update ServiceIntentionSourceIndex to handle peer 2022-06-08 13:53:31 -04:00
Chris S. Kim bb832e2bba
Add SourcePeer fields to relevant Intentions types (#13390) 2022-06-08 13:24:10 -04:00
R.B. Boyer 7423886136
peering: allow protobuf requests to populate the default partition or namespace (#13398) 2022-06-08 11:55:18 -05:00
Dhia Ayachi ec0d267a35
Fix intentions wildcard dest (#13397)
* when enterprise meta are wildcard assume it's a service intention

* fix partition and namespace

* move kind outside the loops

* get the kind check outside the loop and add a comment

Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>
2022-06-08 10:38:55 -04:00
R.B. Boyer edb2e55335
peering: avoid a race between peering establishment and termination (#13389) 2022-06-07 16:29:09 -05:00
Dhia Ayachi 7393374fc0
Egress gtw/intention rpc endpoint (#13354)
* update gateway-services table with endpoints

* fix failing test

* remove unneeded config in test

* rename "endpoint" to "destination"

* more endpoint renaming to destination in tests

* update isDestination based on service-defaults config entry creation

* use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist)

* set unknown state to empty to avoid modifying alot of tests

* fix logic to set the kind correctly on CRUD

* fix failing tests

* add missing tests and fix service delete

* fix failing test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* fix a bug with kind and add relevant test

* fix compile error

* fix failing tests

* add kind to clone

* fix failing tests

* fix failing tests in catalog endpoint

* fix service dump test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* remove duplicate tests

* first draft of destinations intention in connect proxy

* remove ServiceDestinationList

* fix failing tests

* fix agent/consul failing tests

* change to filter intentions in the state store instead of adding a field.

* fix failing tests

* fix comment

* fix comments

* store service kind destination and add relevant tests

* changes based on review

* filter on destinations when querying source match

* change state store API to get an IntentionTarget parameter

* add intentions tests

* add destination upstream endpoint

* fix failing test

* fix failing test and a bug with wildcard intentions

* fix failing test

* Apply suggestions from code review

Co-authored-by: alex <8968914+acpana@users.noreply.github.com>

* add missing test and clarify doc

* fix style

* gofmt intention.go

* fix merge introduced issue

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
Co-authored-by: alex <8968914+acpana@users.noreply.github.com>
Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>
2022-06-07 15:55:02 -04:00
Dhia Ayachi 5ec3274ae5
Egress gtw/connect destination intentions (#13341)
* update gateway-services table with endpoints

* fix failing test

* remove unneeded config in test

* rename "endpoint" to "destination"

* more endpoint renaming to destination in tests

* update isDestination based on service-defaults config entry creation

* use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist)

* set unknown state to empty to avoid modifying alot of tests

* fix logic to set the kind correctly on CRUD

* fix failing tests

* add missing tests and fix service delete

* fix failing test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* fix a bug with kind and add relevant test

* fix compile error

* fix failing tests

* add kind to clone

* fix failing tests

* fix failing tests in catalog endpoint

* fix service dump test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* remove duplicate tests

* first draft of destinations intention in connect proxy

* remove ServiceDestinationList

* fix failing tests

* fix agent/consul failing tests

* change to filter intentions in the state store instead of adding a field.

* fix failing tests

* fix comment

* fix comments

* store service kind destination and add relevant tests

* changes based on review

* filter on destinations when querying source match

* Apply suggestions from code review

Co-authored-by: alex <8968914+acpana@users.noreply.github.com>

* fix style

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* rename destinationType to targetType.

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
Co-authored-by: alex <8968914+acpana@users.noreply.github.com>
Co-authored-by: github-team-consul-core <github-team-consul-core@hashicorp.com>
2022-06-07 15:03:59 -04:00
Alessandro De Blasis b59c19bb06 feat: windows service health checks 2022-06-07 18:27:14 +01:00
R.B. Boyer ab758b7b32
peering: allow mesh gateways to proxy L4 peered traffic (#13339)
Mesh gateways will now enable tcp connections with SNI names including peering information so that those connections may be proxied.

Note: this does not change the callers to use these mesh gateways.
2022-06-06 14:20:41 -05:00
Fulvio d457d8b6ce
UDP check for service stanza #12221 (#12722)
* UDP check for service stanza #12221

* add pass status on timeout condition

* delete useless files

* Update check_test.go

improve comment in test

* fix test

* fix requested changes and update TestRuntimeConfig_Sanitize.golden

* add freeport to TestCheckUDPCritical

* improve comment for CheckUDP struct

* fix requested changes

* fix requested changes

* fix requested changes

* add UDP to proto

* add UDP to proto and add a changelog

* add requested test on agent_endpoint_test.go

* add test for given endpoints

* fix failing tests

* add documentation for udp healthcheck

* regenerate proto using buf

* Update website/content/api-docs/agent/check.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/api-docs/agent/check.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/discovery/checks.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/ecs/configuration-reference.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* Update website/content/docs/ecs/configuration-reference.mdx

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* add debug echo

* add debug circle-ci

* add debug circle-ci bash

* use echo instead of status_stage

* remove debug and status from devtools script and use echo instead

* Update website/content/api-docs/agent/check.mdx

Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>

* fix test

* replace status_stage with status

* replace functions with echo

Co-authored-by: Dhia Ayachi <dhia@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Jared Kirschner <85913323+jkirschner-hashicorp@users.noreply.github.com>
2022-06-06 15:13:19 -04:00
alex bbbc50815a
peering: send leader addr (#13342)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>
2022-06-06 10:00:38 -07:00
Dan Upton b168424398
xds: remove HTTPCheckFetcher dependency (#13366)
This is the OSS portion of enterprise PR 1994

Rather than directly interrogating the agent-local state for HTTP
checks using the `HTTPCheckFetcher` interface, we now rely on the
config snapshot containing the checks.

This reduces the number of changes required to support server xDS
sessions.

It's not clear why the fetching approach was introduced in
931d167ebb.
2022-06-06 15:15:33 +01:00
R.B. Boyer 019aeaa57d
peering: update how cross-peer upstreams and represented in proxycfg and rendered in xds (#13362)
This removes unnecessary, vestigal remnants of discovery chains.
2022-06-03 16:42:50 -05:00
cskh 74158a8aa2
Add isLeader metric to track if a server is a leader (#13304)
CTIA-21: sdd is_leader metric to track if a server is a leader

Co-authored-by: alex <8968914+acpana@users.noreply.github.com>
2022-06-03 13:07:37 -04:00
Freddy 32f125cc0f
Merge pull request #13340 from hashicorp/peering/public-listener 2022-06-02 15:15:29 -06:00
Chris S. Kim 73af9e9737
Fix KVSGet method to handle QueryOptions properly (#13344) 2022-06-02 12:26:18 -04:00
Freddy a09c776645 Update public listener with SPIFFE Validator
Envoy's SPIFFE certificate validation extension allows for us to
validate against different root certificates depending on the trust
domain of the dialing proxy.

If there are any trust bundles from peers in the config snapshot then we
use the SPIFFE validator as the validation context, rather than the
usual TrustedCA.

The injected validation config includes the local root certificates as
well.
2022-06-01 17:06:33 -06:00
freddygv 647c57a416 Add agent cache-type for TrustBundleListByService
There are a handful of changes in this commit:
* When querying trust bundles for a service we need to be able to
  specify the namespace of the service.
* The endpoint needs to track the index because the cache watches use
  it.
* Extracted bulk of the endpoint's logic to a state store function
  so that index tracking could be tested more easily.
* Removed check for service existence, deferring that sort of work to ACL authz
* Added the cache type
2022-06-01 17:05:10 -06:00
freddygv 8b58fa8afe Update assumptions around exported-service config
Given that the exported-services config entry can use wildcards, the
precedence for wildcards is handled as with intentions. The most exact
match is the match that applies for any given service. We do not take
the union of all that apply.

Another update that was made was to reflect that only one
exported-services config entry applies to any given service in a
partition. This is a pre-existing constraint that gets enforced by
the Normalize() method on that config entry type.
2022-06-01 17:03:51 -06:00
Freddy 74ca6406ea
Configure upstream TLS context with peer root certs (#13321)
For mTLS to work between two proxies in peered clusters with different root CAs,
proxies need to configure their outbound listener to use different root certificates
for validation.

Up until peering was introduced proxies would only ever use one set of root certificates
to validate all mesh traffic, both inbound and outbound. Now an upstream proxy
may have a leaf certificate signed by a CA that's different from the dialing proxy's.

This PR makes changes to proxycfg and xds so that the upstream TLS validation
uses different root certificates depending on which cluster is being dialed.
2022-06-01 15:53:52 -06:00
R.B. Boyer 8e530701ce
test: regenerate golden files (#13336)
make envoy-regen
    go test ./agent/config -update
2022-06-01 15:17:03 -05:00
Chris S. Kim fcdd031911
Revert getPathSuffixUnescaped (#13256) 2022-06-01 13:17:14 -04:00
Dan Upton adeabed126
proxycfg: replace direct agent cache usage with interfaces (#13320)
This is the OSS portion of enterprise PRs 1904, 1905, 1906, 1907, 1949,
and 1971.

It replaces the proxycfg manager's direct dependency on the agent cache
with interfaces that will be implemented differently when serving xDS
sessions from a Consul server.
2022-06-01 16:18:06 +01:00
Chris S. Kim 67860bd248
Reimplement fs.FileInfo interface (#13315)
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-06-01 11:09:51 -04:00
Dhia Ayachi 1b779240ae
update gateway-services table with endpoints (#13217)
* update gateway-services table with endpoints

* fix failing test

* remove unneeded config in test

* rename "endpoint" to "destination"

* more endpoint renaming to destination in tests

* update isDestination based on service-defaults config entry creation

* use a 3 state kind to be able to set the kind to unknown (when neither a service or a destination exist)

* set unknown state to empty to avoid modifying alot of tests

* fix logic to set the kind correctly on CRUD

* fix failing tests

* add missing tests and fix service delete

* fix failing test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* fix a bug with kind and add relevant test

* fix compile error

* fix failing tests

* add kind to clone

* fix failing tests

* fix failing tests in catalog endpoint

* fix service dump test

* Apply suggestions from code review

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>

* remove duplicate tests

* rename consts and fix kind when no destination is defined in the service-defaults.

* rename Kind to ServiceKind and change switch to use .(type)

Co-authored-by: Dan Stough <dan.stough@hashicorp.com>
2022-05-31 16:20:12 -04:00
Chris S. Kim f0a9b30174
Update repo to use go:embed (#10996)
Replace bindata packages with stdlib go:embed.
Modernize some uiserver code with newer interfaces introduced in go 1.16 (mainly working with fs.File instead of http.File.
Remove steps that are no longer used from our build files.
Add Github Action to detect differences in agent/uiserver/dist and verify that the files are correct (by compiling UI assets and comparing contents).
2022-05-31 15:33:56 -04:00
Riddhi Shah 1a901953e2
[OSS] Fix merge central config tests (#13309)
Setting the right enterprise meta to fix the merge central config tests.
Re-added the tests that were failing on the OSS to ENT merge.
2022-05-31 12:04:19 -07:00
freddygv 364758ef2f Use embedded SpiffeID for peered upstreams 2022-05-31 09:55:37 -06:00
freddygv c8edec0ab6 Remove intermediate representation of SPIFFE IDs
xDS only ever uses the string representation, so we can avoid passing
around connect.SpiffeIDService objects around.
2022-05-31 09:55:37 -06:00
freddygv 870e7c72d7 Return SPIFFE ID for connect proxies in PeerMeta
Proxies dialing exporting services need to know the SPIFFE ID of
services dialed so that the upstream's SANs can be validated.

This commit attaches the SPIFFE ID to all connect proxies exported over
the peering stream so that they are available to importing clusters.

The data in the SPIFFE ID cannot be re-constructed in peer clusters
because the partition of exported services is overwritten on imports.
2022-05-31 09:55:37 -06:00
Freddy 9427700270
[OSS] Add grpc endpoint to fetch a specific trust bundle (#13292)
Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-05-31 09:54:40 -06:00
Matt Keeler 3795769729
Fix a flaky test (#13282)
At the end of this test we were trying to ensure that updating a service in the local state causes it to re-register the service with the config manager.

The config manager in the same method will also call RegisteredProxies to determine if any need to be removed. This portion of the test is not attempting to verify that behavior.

Because the test is only blocked waiting for the Register event before it can end and assert all the mock expectations were met, we may not see the call to RegisteredProxies. This is especially apparent when tests are run with the race detector.

As we don’t actually care if that method is executed before the end of the test we can simply transition from expecting it to be called exactly once to a 0 or 1 times assertion.
2022-05-27 13:25:08 -04:00
Dan Upton 2427e38839
Enable servers to configure arbitrary proxies from the catalog (#13244)
OSS port of enterprise PR 1822

Includes the necessary changes to the `proxycfg` and `xds` packages to enable
Consul servers to configure arbitrary proxies using catalog data.

Broadly, `proxycfg.Manager` now has public methods for registering,
deregistering, and listing registered proxies — the existing local agent
state-sync behavior has been moved into a separate component that makes use of
these methods.

When an xDS session is started for a proxy service in the catalog, a goroutine
will be spawned to watch the service in the server's state store and
re-register it with the `proxycfg.Manager` whenever it is updated (and clean
it up when the client goes away).
2022-05-27 12:38:52 +01:00
alex fd7a403e11
monitor leadership in peering service (#13257)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2022-05-26 17:55:16 -07:00
Riddhi Shah b6a4271c02
Termporarily disable validation of merge central config response (#13266)
Temporarily disabling the validation of merge central config response since
it is breaking OSS to ENT merging.
A follow up PR will patch the fixes.
2022-05-26 13:49:40 -07:00
Chris S. Kim 6d3bea7129
Add support for streaming CA roots to peers (#13260)
Sender watches for changes to CA roots and sends
them through the replication stream. Receiver saves
CA roots to tablePeeringTrustBundle
2022-05-26 15:24:09 -04:00
Riddhi Shah c78ee7d48f
Remove tests failing on ent (#13255)
Will follow up with the fixed version of these tests that passes in ent.
2022-05-26 10:17:59 -07:00
John Cowen 09c5bac102
Export top-level HCP Enabled go-template variable for UI (#13165)
* Update ui template data to export HCPEnabled at the top level
2022-05-26 17:23:56 +01:00
DanStough 2e2c71d2f2 fix: multiple grpc/http2 services for ingress listeners 2022-05-26 10:43:58 -04:00
Riddhi Shah d8d8c8603e
Add support for merge-central-config query param (#13001)
Adds a new query param merge-central-config for use with the below endpoints:

/catalog/service/:service
/catalog/connect/:service
/health/service/:service
/health/connect/:service

If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).
2022-05-25 13:20:17 -07:00
R.B. Boyer 31526139fd
remove a source of test panics (#13227) 2022-05-25 14:33:00 -05:00