Commit Graph

4122 Commits

Author SHA1 Message Date
Evan Culver e35dd08a63
connect: Upgrade Envoy 1.20 to 1.20.1 (#11895) 2022-01-18 14:35:27 -05:00
Daniel Nephin 71767f1b3e rpc: cleanup exit and blocking condition logic in blockingQuery
Remove some unnecessary comments around query_blocking metric. The only
line that needs any comments in the atomic decrement.

Cleanup the block and return comments and logic. The old comment about
AbandonCh may have been relevant before, but it is expected behaviour
now.

The logic was simplified by inverting the err condition.
2022-01-17 16:59:25 -05:00
Daniel Nephin 72a733bed8 rpc: extract rpcQueryTimeout method
This helps keep the logic in blockingQuery more focused. In the future we
may have a separate struct for RPC queries which may allow us to move this
off of Server.
2022-01-17 16:59:25 -05:00
Daniel Nephin fd0a9fd4f3 rpc: move the index defaulting to setQueryMeta.
This safeguard should be safe to apply in general. We are already
applying it to non-blocking queries that call blockingQuery, so it
should be fine to apply it to others.
2022-01-17 16:59:25 -05:00
Daniel Nephin 4b67d6c18b rpc: add subtests to blockingQuery test 2022-01-17 16:59:25 -05:00
Daniel Nephin f92dc11002 rpc: refactor blocking query
To remove the TODO, and make it more readable.

In general this reduces the scope of variables, making them easier to reason about.
It also introduces more early returns so that we can see the flow from the structure
of the function.
2022-01-17 16:58:47 -05:00
Daniel Nephin f31e0b8b1a
Merge pull request #11661 from hashicorp/dnephin/ca-remove-one-call-to-active-root
ca: remove one call to Provider.ActiveRoot
2022-01-13 16:48:12 -05:00
Kyle Havlovitz 0db874c38b Add virtual IP generation for term gateway backed services 2022-01-12 12:08:49 -08:00
Chris S. Kim 98ea6d1cf1
Fix race with tags (#12041) 2022-01-12 11:24:51 -05:00
Chris S. Kim a0acf9978f
Fix races in anti-entropy tests (#12028) 2022-01-11 14:28:51 -05:00
Mike Morris 1b1a97e8f9
ingress: allow setting TLS min version and cipher suites in ingress gateway config entries (#11576)
* xds: refactor ingress listener SDS configuration

* xds: update resolveListenerSDS call args in listeners_test

* ingress: add TLS min, max and cipher suites to GatewayTLSConfig

* xds: implement envoyTLSVersions and envoyTLSCipherSuites

* xds: merge TLS config

* xds: configure TLS parameters with ingress TLS context from leaf

* xds: nil check in resolveListenerTLSConfig validation

* xds: nil check in makeTLSParameters* functions

* changelog: add entry for TLS params on ingress config entries

* xds: remove indirection for TLS params in TLSConfig structs

* xds: return tlsContext, nil instead of ambiguous err

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>

* xds: switch zero checks to types.TLSVersionUnspecified

* ingress: add validation for ingress config entry TLS params

* ingress: validate listener TLS config

* xds: add basic ingress with TLS params tests

* xds: add ingress listeners mixed TLS min version defaults precedence test

* xds: add more explicit tests for ingress listeners inheriting gateway defaults

* xds: add test for single TLS listener on gateway without TLS defaults

* xds: regen golden files for TLSVersionInvalid zero value, add TLSVersionAuto listener test

* types/tls: change TLSVersion to string

* types/tls: update TLSCipherSuite to string type

* types/tls: implement validation functions for TLSVersion and TLSCipherSuites, make some maps private

* api: add TLS params to GatewayTLSConfig, add tests

* api: add TLSMinVersion to ingress gateway config entry test JSON

* xds: switch to Envoy TLS cipher suite encoding from types package

* xds: fixup validation for TLSv1_3 min version with cipher suites

* add some kitchen sink tests and add a missing struct tag

* xds: check if mergedCfg.TLSVersion is in TLSVersionsWithConfigurableCipherSuites

* xds: update connectTLSEnabled comment

* xds: remove unsued resolveGatewayServiceTLSConfig function

 * xds: add makeCommonTLSContextFromLeafWithoutParams

* types/tls: add LessThan comparator function for concrete values

* types/tls: change tlsVersions validation map from string to TLSVersion keys

* types/tls: remove unused envoyTLSCipherSuites

* types/tls: enable chacha20 cipher suites for Consul agent

* types/tls: remove insecure cipher suites from allowed config

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 and TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 are both explicitly listed as insecure and disabled in the Go source.

Refs https://cs.opensource.google/go/go/+/refs/tags/go1.17.3:src/crypto/tls/cipher_suites.go;l=329-330

* types/tls: add ValidateConsulAgentCipherSuites function, make direct lookup map private

* types/tls: return all unmatched cipher suites in validation errors

* xds: check that Envoy API value matching TLS version is found when building TlsParameters

* types/tls: check that value is found in map before appending to slice in MarshalEnvoyTLSCipherSuiteStrings

* types/tls: cast to string rather than fmt.Printf in TLSCihperSuite.String()

* xds: add TLSVersionUnspecified to list of configurable cipher suites

* structs: update note about config entry warning

* xds: remove TLS min version cipher suite unconfigurable test placeholder

* types/tls: update tests to remove assumption about private map values

Co-authored-by: R.B. Boyer <rb@hashicorp.com>
2022-01-11 11:46:42 -05:00
Dao Thanh Tung 88c7cfa578
URL-encode/decode resource names for HTTP API part 2 (#11957) 2022-01-11 08:52:45 -05:00
Daniel Nephin d57dec5878 ca: remove unnecessary var, and slightly reduce cyclo complexity
`newIntermediate` is always equal to `needsNewIntermediate`, so we can
remove the extra variable and use the original directly.

Also remove the `activeRoot.ID != newActiveRoot.ID` case from an if,
because that case is already checked above, and `needsNewIntermediate` will
already be true in that case.

This condition now reads a lot better:

> Persist a new root if we did not have one before, or if generated a new intermediate.
2022-01-06 16:56:49 -05:00
Daniel Nephin 0de7efb316 ca: remove unused provider.ActiveRoot call
In the previous commit the single use of this storedRoot was removed.

In this commit the original objective is completed. The
Provider.ActiveRoot is being removed because

1. the secondary should get the active root from the Consul primary DC,
   not the provider, so that secondary DCs do not need to communicate
   with a provider instance in a different DC.
2. so that the Provider.ActiveRoot interface can be changed without
   impacting other code paths.
2022-01-06 16:56:48 -05:00
Daniel Nephin d0578c6dfc ca: extract the lookup of the active primary CA
This method had only one caller, which always looked for the active
root. This commit moves the lookup into the method to reduce the logic
in the one caller.

This is being done in preparation for a larger change. Keeping this
separate so it is easier to see.

The `storedRootID != primaryRoots.ActiveRootID` is being removed because
these can never be different.

The `storedRootID` comes from `provider.ActiveRoot`, the
`primaryRoots.ActiveRootID` comes from the store `CARoot` from the
primary. In both cases the source of the data is the primary DC.

Technically they could be different if someone modified the provider
outside of Consul, but that would break many things, so is not a
supported flow.

If these were out of sync because of ordering of events then the
secondary will soon receive an update to `primaryRoots` and everything
will be sorted out again.
2022-01-06 16:56:48 -05:00
Daniel Nephin 7121c78d34 ca: update godoc
To clarify what to expect from the data stored in this field, and the
behaviour of this function.
2022-01-06 16:56:48 -05:00
Daniel Nephin abac8baa5d ca: remove one call to provider.ActiveRoot
ActiveRoot should not be called from the secondary DC, because there
should not be a requirement to run the same Vault instance in a
secondary DC. SignIntermediate is called in a secondary DC, so it should
not call ActiveRoot

We would also like to change the interface of ActiveRoot so that we can
support using an intermediate cert as the primary CA in Consul. In
preparation for making that change I am reducing the number of calls to
ActiveRoot, so that there are fewer code paths to modify when the
interface changes.

This change required a change to the mockCAServerDelegate we use in
tests. It was returning the RootCert for SignIntermediate, but that is
not an accurate fake of production. In production this would also be a
separate cert.
2022-01-06 16:55:50 -05:00
Daniel Nephin eaa084fd41 ca: remove redundant append of an intermediate cert
Immediately above this line we are already appending the full list of
intermediates. The `provider.ActiveIntermediate` MUST be in this list of
intermediates because it must be available to all the other non-leader
Servers.  If it was not in this list of intermediates then any proxy
that received data from a non-leader would have the wrong certs.

This is being removed now because we are planning on changing the
`Provider.ActiveIntermediate` interface, and removing these extra calls ahead of
time helps make that change easier.
2022-01-06 16:55:50 -05:00
Daniel Nephin 11f4cdaa49 ca: only generate a single private key for the whole test case
Using tracing and cpu profiling I found that the majority of the time in
these test cases is spent generating a private key. We really don't need
separate private keys, so we can generate only one and use it for all
cases.

With this change the test runs much faster.
2022-01-06 16:55:50 -05:00
Daniel Nephin b3ffe7ac72 ca: cleanup a test
Fix the name to match the function it is testing

Remove unused code

Fix the signature, instead of returning (error, string) which should be (string, error)
accept a testing.T to emit errors.

Handle the error from encode.
2022-01-06 16:55:49 -05:00
Daniel Nephin 1fd6b16399 ca: use the new leaf signing lookup func in leader metrics 2022-01-06 16:55:49 -05:00
Blake Covarrubias 4bd92921f4
api: Return 404 when deregistering a non-existent check (#11950)
Update the `/agent/check/deregister/` API endpoint to return a 404
HTTP response code when an attempt is made to de-register a check ID
that does not exist on the agent.

This brings the behavior of /agent/check/deregister/ in line with the
behavior of /agent/service/deregister/ which was changed in #10632 to
similarly return a 404 when de-registering non-existent services.

Fixes #5821
2022-01-06 12:38:37 -08:00
Dhia Ayachi 1eac39ae9c
clone the service under lock to avoid a data race (#11940)
* clone the service under lock to avoid a data race

* add change log

* create a struct and copy the pointer to mutate it to avoid a data race

* fix failing test

* revert added space

* add comments, to clarify the data race.
2022-01-06 14:33:06 -05:00
Daniel Nephin 065f6f89fb
Merge pull request #11918 from hashicorp/dnephin/tob-followup
Fix a few small bugs
2022-01-05 18:50:48 -05:00
Daniel Nephin abfc1e4840 snapshot: return the error from replyFn
The only function passed to SnapshotRPC today always returns a nil error, so there's no
way to exercise this bug in practice. This change is being made for correctness so that
it doesn't become a problem in the future, if we ever pass a different function to
SnapshotRPC.
2022-01-05 17:51:03 -05:00
Daniel Nephin 0166b0839c config: correctly capture all errors.
Some calls to multierror.Append were not using the existing b.err, which meant we
were losing all previous errors.
2022-01-05 17:51:03 -05:00
Chris S. Kim 4cd2542a3e
Fix test for ENT (#11946) 2022-01-05 15:18:08 -05:00
Chris S. Kim e4bcaac08c
Fix test for ENT (#11941) 2022-01-05 12:24:44 -05:00
Dhia Ayachi e653f81919
reset `coalesceTimer` to nil as soon as the event is consumed (#11924)
* reset `coalesceTimer` to nil as soon as the event is consumed

* add change log

* refactor to add relevant test.

* fix linter

* Apply suggestions from code review

Co-authored-by: Freddy <freddygv@users.noreply.github.com>

* remove non needed check

Co-authored-by: Freddy <freddygv@users.noreply.github.com>
2022-01-05 12:17:47 -05:00
Mathew Estafanous 0fdd1318e9
Ensure consistency with error-handling across all handlers. (#11599) 2022-01-05 12:11:03 -05:00
Jared Kirschner b393c90ce7 Clarify service and check error messages (use ID)
Error messages related to service and check operations previously included
the following substrings:
- service %q
- check %q

From this error message, it isn't clear that the expected field is the ID for
the entity, not the name. For example, if the user has a service named test,
the error message would read 'Unknown service "test"'. This is misleading -
a service with that *name* does exist, but not with that *ID*.

The substrings above have been modified to make it clear that ID is needed,
not name:
- service with ID %q
- check with ID %q
2022-01-04 11:42:37 -08:00
Jared Kirschner a36ddc31c7
Merge pull request #11335 from littlestar642/url-encoded-args
URL-encode/decode resource names for HTTP API
2022-01-04 14:00:14 -05:00
Chris S. Kim 30550f2c63
testing: Revert assertion for virtual IP flag (#11932) 2022-01-04 11:24:56 -05:00
Jared Kirschner e0ddb9e4c5
Merge pull request #11820 from hashicorp/improve-ui-disabled-api-response
http: improve UI not enabled response message
2022-01-03 12:00:01 -05:00
littlestar642 634c72d22f add path escape and unescape to path params 2022-01-03 08:18:32 -08:00
Daniel Nephin 1683da66b0
Merge pull request #11796 from hashicorp/dnephin/cleanup-test-server
testing: stop using an old version in testServer
2021-12-22 16:04:04 -05:00
freddygv 21f2c2e68d Purge chain if it shouldn't be there 2021-12-13 18:56:44 -07:00
freddygv fe85138453 additional test fixes 2021-12-13 18:56:44 -07:00
freddygv d26b4860fd Account for new upstreams constraint in tests 2021-12-13 18:56:28 -07:00
freddygv 2fe27b748d Check ingress upstreams when gating chain watches 2021-12-13 18:56:28 -07:00
freddygv 6814e84459 Use ptr receiver in all Upstream methods 2021-12-13 18:56:14 -07:00
freddygv 6af9a0d8cf Avoid storing chain without an upstream 2021-12-13 18:56:14 -07:00
freddygv ba12dc215b Clean up chains separately from their watches 2021-12-13 18:56:14 -07:00
freddygv c5c290c503 Validate chains are associated with upstreams
Previously we could get into a state where discovery chain entries were
not cleaned up after the associated watch was cancelled. These changes
add handling for that case where stray chain references are encountered.
2021-12-13 18:56:13 -07:00
freddygv 70d6358426 Store intention upstreams in snapshot 2021-12-13 18:56:13 -07:00
R.B. Boyer 81ea8129d7
proxycfg: ensure all of the watches are canceled if they are cancelable (#11824) 2021-12-13 15:56:17 -06:00
Jared Kirschner f81dd817ff
Merge pull request #11818 from hashicorp/improve-url-not-found-response
http: improve 404 Not Found response message
2021-12-13 16:08:50 -05:00
R.B. Boyer 4aabbe529c
proxycfg: use external addresses in tproxy when crossing partition boundaries (#11823) 2021-12-13 14:34:49 -06:00
Jared Kirschner 2de79abc00 http: improve 404 Not Found response message
When a URL path is not found, return a non-empty message with the 404 status
code to help the user understand what went wrong. If the URL path was not
prefixed with '/v1/', suggest that may be the cause of the problem (which is a
common mistake).
2021-12-13 11:03:25 -08:00
Freddy 85fe875d07
Use anonymousToken when querying by secret ID (#11813)
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: Dan Upton <daniel@floppy.co>

This query has been incorrectly querying by accessor ID since New ACLs
were added. However, the legacy token compat allowed this to continue to
work, since it made a fallback query for the anonymousToken ID.

PR #11184 removed this legacy token query, which means that the query by
accessor ID is now the only check for the anonymous token's existence.

This PR updates the GetBySecret call to use the secret ID of the token.
2021-12-13 10:56:09 -07:00
R.B. Boyer 631c649291
various partition related todos (#11822) 2021-12-13 11:43:33 -06:00
Jared Kirschner 34ea9ae8c9 http: improve UI not enabled response message
Response now clearly indicates:
- the UI is disabled
- how to enable the UI
2021-12-13 08:48:33 -08:00
Kyle Havlovitz b50ef696c6
Merge pull request #11812 from hashicorp/metrics-ui-acls
oss: use wildcard partition in metrics proxy ui endpoint
2021-12-10 16:24:47 -08:00
Kyle Havlovitz 9dcaf0539c
Merge pull request #11798 from hashicorp/vip-goroutine-check
leader: move the virtual IP version check into a goroutine
2021-12-10 15:59:35 -08:00
Kyle Havlovitz 018693b6ee acl: use wildcard partition in metrics proxy ui endpoint 2021-12-10 15:58:17 -08:00
Kyle Havlovitz 80a4489844 state: fix freed VIP table id index 2021-12-10 14:41:45 -08:00
Kyle Havlovitz ecbd3eb2a6 Exit before starting the vip check routine if possible 2021-12-10 14:30:50 -08:00
Daniel Nephin 0a9cb62859 testing: Deprecate functions for creating a server.
These helper functions actually end up hiding important setup details
that should be visible from the test case. We already have a convenient
way of setting this config when calling newTestServerWithConfig.
2021-12-09 20:09:29 -05:00
Daniel Nephin c9a992f5e8 testing: remove old config.Build version
DefaultConfig already sets the version to version.Version, so by removing this
our tests will run with the version that matches the code.
2021-12-09 20:09:29 -05:00
Kyle Havlovitz 04ef1c3fa0 leader: move the virtual IP version check into a goroutine 2021-12-09 17:00:33 -08:00
FFMMM 74eb257b1c
[sync ent] increase segment max limit to 4*64, make configurable (#1424) (#11795)
* commit b6eb27563e747a78b7647d2b5da405e46364cc46
Author: FFMMM <FFMMM@users.noreply.github.com>
Date:   Thu Dec 9 13:53:44 2021 -0800

    increase segment max limit to 4*64, make configurable (#1424)

    Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

* fix: rename ent changelog file

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>
2021-12-09 15:36:11 -08:00
Daniel Nephin f9647ece05
Merge pull request #11780 from hashicorp/dnephin/ca-test-vault-in-secondary
ca: improve test coverage for RenewIntermediate
2021-12-09 12:29:43 -05:00
R.B. Boyer bb75e63eb4
agent: ensure service maintenance checks for matching partitions ahead of other errors (#11788)
This matches behavior in most other agent api endpoints.
2021-12-09 10:05:02 -06:00
Daniel Nephin 4116a143e0 fix misleading errors on vault shutdown 2021-12-08 18:42:52 -05:00
Daniel Nephin 968aeff1bb ca: prune some unnecessary lookups in the tests 2021-12-08 18:42:52 -05:00
Daniel Nephin 305655a8b1 ca: remove duplicate WaitFor function 2021-12-08 18:42:52 -05:00
Daniel Nephin 1dec6bb815 ca: fix flakes in RenewIntermediate tests
I suspect one problem was that we set structs.IntermediateCertRenewInterval to 1ms, which meant
that in some cases the intermediate could renew before we stored the original value.

Another problem was that the 'wait for intermediate' loop was calling the provider.ActiveIntermediate,
but the comparison needs to use the RPC endpoint to accurately represent a user request. So
changing the 'wait for' to use the state store ensures we don't race.

Also moves the patching into a separate function.

Removes the addition of ca.CertificateTimeDriftBuffer as part of calculating halfTime. This was added
in a previous commit to attempt to fix the flake, but it did not appear to fix the problem. Adding the
time here was making the tests fail when using the shared patch
function. It's not clear to me why, but there's no reason we should be
including this time in the halfTime calculation.
2021-12-08 18:42:52 -05:00
Daniel Nephin 2e4e8bd791 ca: improve RenewIntermediate tests
Use the new verifyLearfCert to show the cert verifies with intermediates
from both sources. This required using the RPC interface so that the
leaf pem was constructed correctly.

Add IndexedCARoots.Active since that is a common operation we see in a
few places.
2021-12-08 18:42:52 -05:00
Daniel Nephin a4ba1f348d ca: add a test for Vault in secondary DC 2021-12-08 18:42:51 -05:00
Daniel Nephin a5d9b1d322 ca: Add CARoots.Active method
Which will be used in the next commit.
2021-12-08 18:41:51 -05:00
R.B. Boyer 5f5720837b
acl: ensure that the agent recovery token is properly partitioned (#11782) 2021-12-08 17:11:55 -06:00
Daniel Nephin f72e285fe8
Merge pull request #11721 from hashicorp/dnephin/ca-export-fsm-operation
ca: use the real FSM operation in tests
2021-12-08 17:49:00 -05:00
Daniel Nephin 214dcf8d0d ca: use the real FSM operation in tests
Previously we had a couple copies that reproduced the FSM operation.
These copies introduce risk that the test does not accurately match
production.

This PR removes the test versions of the FSM operation, and exports the
real production FSM operation so that it can be used in tests.

The consul provider tests did need to change because of this. Previously
we would return a hardcoded value of 2, but in production this value is
always incremented.
2021-12-08 17:29:44 -05:00
R.B. Boyer 592ac8f96a
test: test server should auto cleanup (#11779) 2021-12-08 13:26:06 -06:00
Evan Culver 7a365fa0da
rpc: Unset partition before forwarding to remote datacenter (#11758) 2021-12-08 11:02:14 -08:00
Daniel Nephin dccd3f5806 Merge remote-tracking branch 'origin/main' into serve-panic-recovery 2021-12-07 16:30:41 -05:00
Dan Upton 7efab269c0
Rename `Master` and `AgentMaster` fields in config protobuf (#11764) 2021-12-07 19:59:38 +00:00
Chris S. Kim f8f8580ab2
Godocs updates for catalog endpoints (#11716) 2021-12-07 10:18:28 -05:00
Mathew Estafanous 0a9621ec7a
Transition all endpoint tests in agent_endpoint_test.go to go through ServeHTTP (#11499) 2021-12-07 09:44:03 -05:00
Dan Upton 205ce9a69d
Remove references to "master" ACL tokens in tests (#11751) 2021-12-07 12:48:50 +00:00
Dan Upton 7fe81171d9
Rename `ACLMasterToken` => `ACLInitialManagementToken` (#11746) 2021-12-07 12:39:28 +00:00
Dan Upton 3a91815169
agent/token: rename `agent_master` to `agent_recovery` (internally) (#11744) 2021-12-07 12:12:47 +00:00
R.B. Boyer 9315a9812f return the max 2021-12-06 15:36:52 -06:00
freddygv 60fe5f75bb Remove support for failover to partition
Failing over to a partition is more siimilar to failing over to another
datacenter than it is to failing over to a namespace. In a future
release we should update how localities for failover are specified. We
should be able to accept a list of localities which can include both
partition and datacenter.
2021-12-06 12:32:24 -07:00
freddygv 5c1f7aa372 Allow cross-partition references in disco chain
* Add partition fields to targets like service route destinations
* Update validation to prevent cross-DC + cross-partition references
* Handle partitions when reading config entries for disco chain
* Encode partition in compiled targets
2021-12-06 12:32:19 -07:00
R.B. Boyer b1605639fc
light refactors to support making partitions and serf-based wan federation are mutually exclusive (#11755) 2021-12-06 13:18:02 -06:00
R.B. Boyer e20e6348dd
areas: make the gRPC server tracker network area aware (#11748)
Fixes a bug whereby servers present in multiple network areas would be
properly segmented in the Router, but not in the gRPC mirror. This would
lead servers in the current datacenter leaving from a network area
(possibly during the network area's removal) from deleting their own
records that still exist in the standard WAN area.

The gRPC client stack uses the gRPC server tracker to execute all RPCs,
even those targeting members of the current datacenter (which is unlike
the net/rpc stack which has a bypass mechanism).

This would manifest as a gRPC method call never opening a socket because
it would block forever waiting for the current datacenter's pool of
servers to be non-empty.
2021-12-06 09:55:54 -06:00
Freddy a725f06c83
Merge pull request #11739 from hashicorp/ap/exports-rename 2021-12-06 08:20:50 -07:00
freddygv e91509383f Clean up additional refs to partition exports 2021-12-04 15:16:40 -07:00
freddygv ed6076db26 Rename partition-exports to exported-services
Using a name less tied to partitions gives us more flexibility to use
this config entry in OSS for exports between datacenters/meshes.
2021-12-03 17:47:31 -07:00
freddygv f5b25401b3 Update intention topology to use new table 2021-12-03 17:28:31 -07:00
freddygv 55970c6ccd Avoid updating default decision from wildcard ixn
Given that we do not allow wildcard partitions in intentions, no one ixn
can override the DefaultAllow setting. Only the default ACL policy
applies across all partitions.
2021-12-03 17:28:12 -07:00
freddygv 497aab669f Add a new table to query service names by kind
This table purposefully does not index by partition/namespace. It's a
global view into all service names.

This table is intended to replace the current serviceListTxn watch in
intentionTopologyTxn. For cross-partition transparent proxying we need
to be able to calculate upstreams from intentions in any partition. This
means that the existing serviceListTxn function is insufficient since
it's scoped to a partition.

Moving away from that function is also beneficial because it watches the
main "services" table, so watchers will wake up when any instance is
registered or deregistered.
2021-12-03 17:28:12 -07:00
freddygv e7a7042c69 Update listener generation to account for consul VIP 2021-12-03 17:27:56 -07:00
Freddy f032d6ef05
Merge pull request #11680 from hashicorp/ap/partition-exports-oss 2021-12-03 16:57:50 -07:00
Dan Upton 3b9dfca88d
internal: support `ResultsFilteredByACLs` flag/header (#11643) 2021-12-03 23:04:24 +00:00
Dan Upton c8204330ed
query: support `ResultsFilteredByACLs` in query list endpoint (#11620) 2021-12-03 23:04:09 +00:00
Dhia Ayachi ce326b6074
port oss changes (#11736) 2021-12-03 17:23:55 -05:00
Freddy e246defb6c
Merge pull request #11720 from hashicorp/bbolt 2021-12-03 14:44:36 -07:00
Dan Upton 047aa2ffb0
fedstate: support `ResultsFilteredByACLs` in `ListMeshGateways` endpoint (#11644) 2021-12-03 20:56:55 +00:00
Dan Upton 361d9c2862
catalog: support `ResultsFilteredByACLs` flag/header (#11594) 2021-12-03 20:56:14 +00:00
Dan Upton 4c0956c03a
coordinate: support `ResultsFilteredByACLs` flag/header (#11617) 2021-12-03 20:51:02 +00:00
Dan Upton bf1e2ca551
sessions: support `ResultsFilteredByACLs` flag/header (#11606) 2021-12-03 20:43:43 +00:00
Dan Upton d92f0d84c6
txn: support `ResultsFilteredByACLs` flag in `Read` endpoint (#11632) 2021-12-03 20:41:03 +00:00
Dan Upton 547aa219ea
agent: support `X-Consul-Results-Filtered-By-ACLs` header in agent-local endpoints (#11610) 2021-12-03 20:36:28 +00:00
Dhia Ayachi 86159c6ed8
sessions partitioning tests (#11734)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add `singleValueID` implementation assertions

* partition `tableSessions` table

* fix sessions to use UUID and fix prefix index

* fix oss build

* clean up unused functions

* fix oss compilation

* add a partition indexer for sessions

* Fix oss to not have partition index

* fix oss tests

* remove unused operations_ent.go and operations_oss.go func

* remove unused const

* convert `IndexID` of `session_checks` table

* convert `indexSession` of `session_checks` table

* convert `indexNodeCheck` of `session_checks` table

* partition `indexID` and `indexSession` of `tableSessionChecks`

* fix oss linter

* fix review comments

* remove partition for Checks as it's always use the session partition

* fix tests

* fix tests

* do not namespace nodeChecks index

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-12-03 15:36:07 -05:00
Dan Upton c314be2ff9
intention: support `ResultsFilteredByACLs` flag/header (#11612) 2021-12-03 20:35:54 +00:00
Mark Anderson a89ffba2d4
Cross port of ent #1383 (#11726)
Cross port of ent #1383 "Reject non-default datacenter when making partitioned ACLs"

On the OSS side this is a minor refactor to add some more checks that are only applicable to enterprise code.

Signed-off-by: Mark Anderson <manderson@hashicorp.com>
2021-12-03 10:20:25 -08:00
Dan Upton 599a4d6619
config: support `ResultsFilteredByACLs` in list/list all endpoints (#11621) 2021-12-03 17:39:47 +00:00
Dan Upton c4c68915c9
event: support `X-Consul-Results-Filtered-By-ACLs` header in list (#11616) 2021-12-03 17:38:59 +00:00
Dan Upton 474ef7cc1f
kv: support `ResultsFilteredByACLs` in list/list keys (#11593) 2021-12-03 17:31:48 +00:00
Dan Upton cf1bd585f6
health: support `ResultsFilteredByACLs` flag/header (#11602) 2021-12-03 17:31:32 +00:00
Dan Upton 1e47e3c82b
Groundwork for exposing when queries are filtered by ACLs (#11569) 2021-12-03 17:11:26 +00:00
Kyle Havlovitz 0546bbe08a dns: add endpoint for querying service virtual IPs 2021-12-02 16:40:28 -08:00
Kyle Havlovitz 6f34a4f777
Merge pull request #11724 from hashicorp/service-virtual-ips
oss: add virtual IP generation for connect services
2021-12-02 16:16:57 -08:00
Kyle Havlovitz 4f2cfee4b0 consul: add virtual IP generation for connect services 2021-12-02 15:42:47 -08:00
R.B. Boyer c46f9f9f31
agent: add variation of force-leave that exclusively works on the WAN (#11722)
Fixes #6548
2021-12-02 17:15:10 -06:00
Matt Keeler c7a94843ee Emit raft-boltdb metrics 2021-12-02 16:56:15 -05:00
Daniel Nephin e47cecc653 config: add NoFreelistSync option
# Conflicts:
#	agent/config/testdata/TestRuntimeConfig_Sanitize-enterprise.golden
#	agent/consul/server.go
2021-12-02 16:56:15 -05:00
Matt Keeler 42a5635bc3 Use raft-boltdb/v2 2021-12-02 16:56:15 -05:00
Daniel Nephin 17a2d14d49 ca: set the correct SigningKeyID after config update with Vault provider
The test added in this commit shows the problem. Previously the
SigningKeyID was set to the RootCert not the local leaf signing cert.

This same bug was fixed in two other places back in 2019, but this last one was
missed.

While fixing this bug I noticed I had the same few lines of code in 3
places, so I extracted a new function for them.

There would be 4 places, but currently the InitializeCA flow sets this
SigningKeyID in a different way, so I've left that alone for now.
2021-12-02 16:07:11 -05:00
Daniel Nephin 96f95889db
Merge pull request #11713 from hashicorp/dnephin/ca-test-names
ca: make test naming consistent
2021-12-02 16:05:42 -05:00
Daniel Nephin ff4581092e
Merge pull request #11671 from hashicorp/dnephin/ca-fix-storing-vault-intermediate
ca: fix storing the leaf signing cert with Vault provider
2021-12-02 16:02:24 -05:00
Daniel Nephin 81afb208ac
Merge pull request #11677 from hashicorp/dnephin/freeport-interface
sdk: use t.Cleanup in freeport and remove unnecessary calls
2021-12-02 15:58:41 -05:00
Daniel Nephin 447097b166 ca: make test naming consistent
While working on the CA system it is important to be able to run all the
tests related to the system, without having to wait for unrelated tests.
There are many slow and unrelated tests in agent/consul, so we need some
way to filter to only the relevant tests.

This PR renames all the CA system related tests to start with either
`TestCAMananger` for tests of internal operations that don't have RPC
endpoint, or `TestConnectCA` for tests of RPC endpoints. This allows us
to run all the test with:

    go test -run 'TestCAMananger|TestConnectCA' ./agent/consul

The test naming follows an undocumented convention of naming tests as
follows:

    Test[<struct name>_]<function name>[_<test case description>]

I tried to always keep Primary/Secondary at the end of the description,
and _Vault_ has to be in the middle because of our regex to run those
tests as a separate CI job.

You may notice some of the test names changed quite a bit. I did my best
to identify the underlying method being tested, but I may have been
slightly off in some cases.
2021-12-02 14:57:09 -05:00
FFMMM 384d497f26
add MustRevalidate flag to connect_ca_leaf cache type; always use on non-blocking queries (#11693)
* always use MustRevalidate on non-blocking queries for connect ca leaf

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

* Update agent/agent_endpoint_test.go

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>

* pr feedback

Signed-off-by: FFMMM <FFMMM@users.noreply.github.com>

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
2021-12-02 11:32:15 -08:00
Daniel Nephin 28a8a64019 ca: make getLeafSigningCertFromRoot safer
As a method on the struct type this would not be safe to call without first checking
c.isIntermediateUsedToSignLeaf.

So for now, move this logic to the CAMananger, so that it is always correct.
2021-12-02 12:42:49 -05:00
Daniel Nephin b29faa3e50 ca: fix stored CARoot representation with Vault provider
We were not adding the local signing cert to the CARoot. This commit
fixes that bug, and also adds support for fixing existing CARoot on
upgrade.

Also update the tests for both primary and secondary to be more strict.
Check the SigningKeyID is correct after initialization and rotation.
2021-12-02 12:42:49 -05:00
Dan Upton bf56a2c495
Rename `agent_master` ACL token in the API and CLI (#11669) 2021-12-02 17:05:27 +00:00
Dan Upton d8afd2f6c8
Rename `master` and `agent_master` ACL tokens in the config file format (#11665) 2021-12-01 21:08:14 +00:00
Chris S. Kim 54e4d1b7b2
ENT to OSS sync (#11703) 2021-12-01 14:56:10 -05:00
R.B. Boyer db91cbf484
auto-config: ensure the feature works properly with partitions (#11699) 2021-12-01 13:32:34 -06:00
Daniel Nephin 32ef9c5d5c ca: add some godoc and func for finding leaf signing cert
This will be used in a follow up commit.
2021-11-30 18:36:41 -05:00
Daniel Nephin 4185045a7f sdk/freeport: rename Port to GetOne
For better consistency with GetN
2021-11-30 17:32:41 -05:00
Chris S. Kim 56fab21582
Refactor test helper (#11689)
Allow custom ACL root tokens to be passed
2021-11-30 13:22:07 -05:00
Chris S. Kim 36246c5791
acl: Fill authzContext from token in Coordinate endpoints (#11688) 2021-11-30 13:17:41 -05:00
freddygv dd662d7058 Move ent config test to ent file 2021-11-29 12:15:17 -07:00
freddygv 5e1f7b7c36 Prevent partition-exports entry from OSS usage
Validation was added on the config entry kind since that is called when
validating config entries to bootstrap via agent configuration and when
applying entries via the config RPC endpoint.
2021-11-29 11:24:16 -07:00
Daniel Nephin e8312d6b5a testing: remove unnecessary calls to freeport
Previously we believe it was necessary for all code that required ports
to use freeport to prevent conflicts.

https://github.com/dnephin/freeport-test shows that it is actually save
to use port 0 (`127.0.0.1:0`) as long as it is passed directly to
`net.Listen`, and the listener holds the port for as long as it is
needed.

This works because freeport explicitly avoids the ephemeral port range,
and port 0 always uses that range. As you can see from the test output
of https://github.com/dnephin/freeport-test, the two systems never use
overlapping ports.

This commit converts all uses of freeport that were being passed
directly to a net.Listen to use port 0 instead. This allows us to remove
a bit of wrapping we had around httptest, in a couple places.
2021-11-29 12:19:43 -05:00
Daniel Nephin d795a73f78 testing: use the new freeport interfaces 2021-11-27 15:39:46 -05:00
Daniel Nephin 56f9238d15 go-sso: remove returnFunc now that freeport handles return 2021-11-27 15:29:38 -05:00
Daniel Nephin 8c7475d95e sdk: add freeport functions that use t.Cleanup 2021-11-27 15:04:43 -05:00
Daniel Nephin 59204598c8 ca: clean up unnecessary raft.Apply response checking
In d2ab767fef raftApply was changed to handle this check in
a single place, instad of having every caller check it. It looks like these few places
were missed when I did that clean up.

This commit removes the remaining resp.(error) checks, since they are all no-ops now.
2021-11-26 17:57:55 -05:00
Daniel Nephin 52f0853ff9
Merge pull request #11339 from hashicorp/dnephin/ca-manager-isolate-secondary-2
ca: reduce use of state in the secondary
2021-11-26 14:41:45 -05:00
Daniel Nephin 91a0c25932 ca: remove state check in secondarySetPrimaryRoots
This function is only ever called from operations that have already acquired the state lock, so checking
the value of state can never fail.

This change is being made in preparation for splitting out a separate type for the secondary logic. The
state can't easily be shared, so really only the expored top-level functions should acquire the 'state lock'.
2021-11-26 14:14:47 -05:00
Daniel Nephin f1944458e4 ca: remove actingSecondaryCA
This commit removes the actingSecondaryCA field, and removes the stateLock around it. This field
was acting as a proxy for providerRoot != nil, so replace it with that check instead.

The two methods which called secondarySetCAConfigured already set the state, so checking the
state again at this point will not catch runtime errors (only programming errors, which we can catch with tests).
In general, handling state transitions should be done on the "entrypoint" methods where execution starts, not
in every internal method.

This is being done to remove some unnecessary references to c.state, in preparations for extracting
types for primary/secondary.
2021-11-26 14:14:47 -05:00
Daniel Nephin b92084b8e8 ca: reduce consul provider backend interface a bit
This makes it easier to fake, which will allow me to use the ConsulProvider as
an 'external PKI' to test a customer setup where the actual root CA is not
the root we use for the Consul CA.

Replaces a call to the state store to fetch the clusterID with the
clusterID field already available on the built-in provider.
2021-11-25 11:46:06 -05:00
Dhia Ayachi 3820e09a47
Partition/kv indexid sessions (#11639)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* partition `tableSessions` table

* fix sessions to use UUID and fix prefix index

* fix oss build

* clean up unused functions

* fix oss compilation

* add a partition indexer for sessions

* Fix oss to not have partition index

* fix oss tests

* remove unused operations_ent.go and operations_oss.go func

* convert `indexNodeCheck` of `session_checks` table

* partition `indexID` and `indexSession` of `tableSessionChecks`

* remove partition for Checks as it's always use the session partition

* partition sessions index id table

* fix rebase issues

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-11-24 11:34:36 -05:00
Dhia Ayachi bb83624950
Partition session checks store (#11638)
* state: port KV and Tombstone tables to new pattern

* go fmt'ed

* handle wildcards for tombstones

* Fix graveyard ent vs oss

* fix oss compilation error

* add partition to tombstones and kv state store indexes

* refactor to use `indexWithEnterpriseIndexable`

* Apply suggestions from code review

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>

* add `singleValueID` implementation assertions

* partition `tableSessions` table

* fix sessions to use UUID and fix prefix index

* fix oss build

* clean up unused functions

* fix oss compilation

* add a partition indexer for sessions

* Fix oss to not have partition index

* fix oss tests

* remove unused operations_ent.go and operations_oss.go func

* remove unused const

* convert `IndexID` of `session_checks` table

* convert `indexSession` of `session_checks` table

* convert `indexNodeCheck` of `session_checks` table

* partition `indexID` and `indexSession` of `tableSessionChecks`

* fix oss linter

* fix review comments

* remove partition for Checks as it's always use the session partition

Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2021-11-24 09:10:38 -05:00
Chris S. Kim 2350e7e56a
cleanup: Clarify deprecated legacy intention endpoints (#11635) 2021-11-23 19:32:18 -05:00