Commit Graph

17460 Commits

Author SHA1 Message Date
Jared Kirschner bf4d23a9e8
Merge pull request #13006 from hashicorp/docs/http-api-breakout-path-parameters
HTTP API Docs: Separate path parameters from query parameters
2022-05-25 18:20:40 -04:00
Jared Kirschner 49a7e7086c docs: split HTTP API params into sections by type
Path parameters, query parameters, and request body parameters are now shown in
separate sections rather than combined into one general parameters section.
This makes it much easier to understand quickly where a parameter should be
provided.
2022-05-25 14:45:47 -07:00
Riddhi Shah d8d8c8603e
Add support for merge-central-config query param (#13001)
Adds a new query param merge-central-config for use with the below endpoints:

/catalog/service/:service
/catalog/connect/:service
/health/service/:service
/health/connect/:service

If set on the request, the response will include a fully resolved service definition which is merged with the proxy-defaults/global and service-defaults/:service config entries (on-demand style). This is useful to view the full service definition for a mesh service (connect-proxy kind or gateway kind) which might not be merged before being written into the catalog (example: in case of services in the agentless model).
2022-05-25 13:20:17 -07:00
R.B. Boyer 31526139fd
remove a source of test panics (#13227) 2022-05-25 14:33:00 -05:00
R.B. Boyer a85b8a4705
api: ensure peering API endpoints do not use protobufs (#13204)
I noticed that the JSON api endpoints for peerings json encodes protobufs directly, rather than converting them into their `api` package equivalents before marshal/unmarshaling them.

I updated this and used `mog` to do the annoying part in the middle. 

Other changes:
- the status enum was converted into the friendlier string form of the enum for readability with tools like `curl`
- some of the `api` library functions were slightly modified to match other similar endpoints in UX (cc: @ndhanushkodi )
- peeringRead returns `nil` if not found
- partitions are NOT inferred from the agent's partition (matching 1.11-style logic)
2022-05-25 13:43:35 -05:00
Kyle Schochenmaier 72a1aea56c
update docs for single-dc-multi-k8s install (#13008)
* update docs for single-dc-multi-k8s install

Co-authored-by: David Yu <dyu@hashicorp.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
2022-05-25 11:34:56 -07:00
Michael Wilkerson e55a269601
Nia/docs 0.6.0 (#13107)
* updated docs
- added docs for start command
- deprecated running without a command
- added instructions for autocomplete setup

* addressed review comments

* addressed review comments

* addressed review comments

* docs/nia: Terraform Cloud agent support

- Add agent as a supported execution mode
- Add terraform_cloud_workspace configuration
- Deprecate existing terraform_version config

* license block docs

* added HCP Consul to compatibility

* added HCP instructions

* addressed review comments

* added new auto-retrieval behavior to license docs

* addressed review comments

* addressed review comments

* Apply suggestions from code review

* Apply suggestions from code review

* updated docs

* updated docs

* Apply suggestions from code review

Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

* fixed heading types

* fixed heading types

* update docs

* docs/nia: Add service registration configurations

* docs/nia: Style guide updates

* docs/nia: Update with beta docs feedback

* docs/nia: Update license config formatting

Other top-level blocks aren't included in the list of global config options,
so removed the liciense entry.

* docs/nia: Add auto-retrieval section to license page

* docs/nia: Separate column for HCP Consul support

* docs/nia: Compatiblity version upper bounds

* docs/nia: Fix broken links

* docs/nia: Style guide fixes

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>

* Remove RequestId field from cts health api docs.

* docs/nia - Update CTS service id format (#13125)

* docs/nia: Convert Consul config to table format

* docs/nia: Add ACL token policy requirements

* update docs (#13174)

* docs/nia: Fix ca_path, key default, and some links

* docs/nia: Add CTS service address config

* Update website/content/docs/nia/cli/index.mdx

* docs/nia: update for 0.6 GA (#13191)



Co-authored-by: devarshishah3 <devarshishah3@gmail.com>
Co-authored-by: Michael Wilkerson <62034708+wilkermichael@users.noreply.github.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>

Co-authored-by: Melissa Kam <mkam@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Melissa Kam <3768460+mkam@users.noreply.github.com>
Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
Co-authored-by: Derek Menteer <derek.menteer@hashicorp.com>
Co-authored-by: lornasong <lornasong@users.noreply.github.com>
Co-authored-by: hashi-derek <105233703+hashi-derek@users.noreply.github.com>
Co-authored-by: devarshishah3 <devarshishah3@gmail.com>
2022-05-25 14:23:43 -04:00
R.B. Boyer 1a8834e1c8
peering: replicate expected SNI, SPIFFE, and service protocol to peers (#13218)
The importing peer will need to know what SNI and SPIFFE name
corresponds to each exported service. Additionally it will need to know
at a high level the protocol in use (L4/L7) to generate the appropriate
connection pool and local metrics.

For replicated connect synthetic entities we edit the `Connect{}` part
of a `NodeService` to have a new section:

    {
      "PeerMeta": {
        "SNI": [
          "web.default.default.owt.external.183150d5-1033-3672-c426-c29205a576b8.consul"
        ],
        "SpiffeID": [
          "spiffe://183150d5-1033-3672-c426-c29205a576b8.consul/ns/default/dc/dc1/svc/web"
        ],
        "Protocol": "tcp"
      }
    }

This data is then replicated and saved as-is at the importing side. Both
SNI and SpiffeID are slices for now until I can be sure we don't need
them for how mesh gateways will ultimately work.
2022-05-25 12:37:44 -05:00
R.B. Boyer 35371ad697
build: re-add proto-tools dep to make proto (#13223) 2022-05-25 12:24:55 -05:00
Evan Culver a82b5f3933
update main to reflect 1.13.0-dev (#13192) 2022-05-25 09:06:36 -07:00
Evan Culver 2abccd78b4
Update CHANGELOG to mention removal of Envoy 1.17.4 and 1.18.6 (#13207) 2022-05-25 08:57:01 -07:00
R.B. Boyer be631ebdce
peering: disable requirement for mesh gateways initially (#13213) 2022-05-25 10:13:23 -05:00
Kyle Havlovitz 0ed9ff8ef7
Merge pull request #13143 from hashicorp/envoy-connection-limit
Add connection limit setting to service defaults
2022-05-25 07:48:50 -07:00
John Cowen a61e5cc08b
ui: Icon related fixups (#13183)
* ui: Use new icon-size and icon-color for popover-menus

* Remove the default currentColor plus add some more defaults

* Undo transparency overwrites now we don't need them

* Fixup discochain icons

* Undo a default icon rule for vert align

* Fixup expanded icon for meatball popovers

* Fixup intention permission labels/badges/icons

* Remove different res icon

* Remove icon resolutions
2022-05-25 14:28:42 +01:00
Kyle Havlovitz f2fbe8aec9 Fix proto lint errors after version bump 2022-05-24 18:44:54 -07:00
Michele Degges bfe7f0ad63
[CI-only] Update tagging for dev_tags (#13199)
Remove the hardcoded `-dev` suffix from dev_tags, which is causing tags to be in the format `1.12.0-dev-dev` instead of just `1.12.0-dev`. I'll clean up the old tags before making the dockerhub repo public, which will be available https://hub.docker.com/r/hashicorppreview/consul
2022-05-24 15:23:01 -07:00
Kyle Havlovitz dbed8ae10b Specify go_package explicitly 2022-05-24 10:22:53 -07:00
cskh 8712a088b1
fix: non-leader agents return 404 on Get Intention exact api (#13179)
* fix: non-leader agents return 404 on Get Intention exact api

- rpc call method appends extra error message, so change == to
  "Strings.Contains"

Co-authored-by: Chris S. Kim <ckim@hashicorp.com>
2022-05-24 13:21:15 -04:00
Kyle Havlovitz 4bc6c23357 Add connection limit setting to service defaults 2022-05-24 10:13:38 -07:00
Jasmine W 4c04d70fb6
Merge pull request #13188 from hashicorp/ui/bugfix/permission-alignment
ui: Alignment of L7 permissions
2022-05-24 12:27:30 -04:00
DanStough 817449041d chore(test): Update bats version 2022-05-24 11:56:08 -04:00
DanStough 147fd96d97 feat: add endpoint struct to ServiceConfigEntry 2022-05-24 11:56:08 -04:00
alex 876f3bb971
peering: expose IsLeader, hung up on dialer if follower (#13164)
Signed-off-by: acpana <8968914+acpana@users.noreply.github.com>

Co-authored-by: R.B. Boyer <4903+rboyer@users.noreply.github.com>
2022-05-23 11:30:58 -07:00
Jasmine W 30f6be389a ui: Center alignment of L7 permissions 2022-05-23 13:21:58 -04:00
R.B. Boyer 9e1f362499
ci: github actions run with TERM=dumb and can't do tput commands (#13187) 2022-05-23 11:43:14 -05:00
R.B. Boyer 7b72ddfb60
build: wire up remaining 5 helper tools into the same auto-install logic used for protobuf tools (#13184) 2022-05-23 10:50:45 -05:00
Matt Keeler 26f4ea3f01
Migrate from `protoc` to `buf` (#12841)
* Install `buf` instead of `protoc`
* Created `buf.yaml` and `buf.gen.yaml` files in the two proto directories to control how `buf` generates/lints proto code.
* Invoke `buf` instead of `protoc`
* Added a `proto-format` make target.
* Committed the reformatted proto files.
* Added a `proto-lint` make target.
* Integrated proto linting with CI
* Fixed tons of proto linter warnings.
* Got rid of deprecated builtin protoc-gen-go grpc plugin usage. Moved to direct usage of protoc-gen-go-grpc.
* Unified all proto directories / go packages around using pb prefixes but ensuring all proto packages do not have the prefix.
2022-05-23 10:37:52 -04:00
cskh c986940fda
Upgrade golangci-lint for go v1.18 (#13176) 2022-05-23 10:26:45 -04:00
R.B. Boyer 21bb0eef4a
test: fix flaky test TestEventBufferFuzz (#13175) 2022-05-23 09:22:30 -05:00
Matt Keeler d0fdf22f83
Fix tests broken in #13173 (#13178)
I changed the error type returned in a situation but didn’t update the tests to expect that error.
2022-05-23 10:00:06 -04:00
Matt Keeler 3c1e17cbd5
Fix flaky tests in the agent/grpc/public/services/serverdiscovery package (#13173)
Occasionally we had seen the TestWatchServers_ACLToken_PermissionDenied be flagged as flaky in circleci. This change should fix that.

Why it fixes it is complicated. The test was failing with a panic when a mocked ACL Resolver was being called more times than expected. I struggled for a while to determine how that could be. This test should call authorize once and only once and the error returned should cause the stream to be terminated and the error returned to the gRPC client. Another oddity was no amount of running this test locally seemed to be able to reproduce the issue. I ran the test hundreds of thousands of time and it always passed.

It turns out that there is nothing wrong with the test. It just so happens that the panic from unexpected invocation of a mocked call happened during the test but was caused by a previous test (specifically the TestWatchServers_StreamLifecycle test)

The stream from the previous test remained open after all the test Cleanup functions were run and it just so happened that when the EventPublisher eventually picked up that the context was cancelled during cleanup, it force closes all subscriptions which causes some loops to be re-entered and the streams to be reauthorized. Its that looping in response to forced subscription closures that causes the mock to eventually panic. All the components, publisher, server, client all operate based on contexts. We cancel all those contexts but there is no syncrhonous way to know when they are stopped.

We could have implemented a syncrhonous stop but in the context of an actual running Consul, context cancellation + async stopping is perfectly fine. What we (Dan and I) eventually thought was that the behavior of grpc streams such as this when a server was shutting down wasn’t super helpful. What we would want is for a client to be able to distinguish between subscription closed because something may have changed requiring re-authentication and subscription closed because the server is shutting down. That way we can send back appropriate error messages to detail that the server is shutting down and not confuse users with potentially needing to resubscribe.

So thats what this PR does. We have introduced a shutting down state to our event subscriptions and the various streaming gRPC services that rely on the event publisher will all just behave correctly and actually stop the stream (not attempt transparent reauthorization) if this particular error is the one we get from the stream. Additionally the error that gets transmitted back through gRPC when this does occur indicates to the consumer that the server is going away. That is more helpful so that a client can then attempt to reconnect to another server.
2022-05-23 08:59:13 -04:00
Evan Culver 7ccbb3489b
ci: name unnamed GHA workflows (#13162) 2022-05-20 16:06:35 -07:00
R.B. Boyer bbcb1fa805
agent: allow for service discovery queries involving peer name to use streaming (#13168) 2022-05-20 15:27:01 -05:00
Dan Upton d7f8a8e4ef
proxycfg: remove dependency on `cache.UpdateEvent` (#13144)
OSS portion of enterprise PR 1857.

This removes (most) references to the `cache.UpdateEvent` type in the
`proxycfg` package.

As we're going to be direct usage of the agent cache with interfaces that
can be satisfied by alternative server-local datasources, it doesn't make
sense to depend on this type everywhere anymore (particularly on the
`state.ch` channel).

We also plan to extract `proxycfg` out of Consul into a shared library in
the future, which would require removing this dependency.

Aside from a fairly rote find-and-replace, the main change is that the
`cache.Cache` and `health.Client` types now accept a callback function
parameter, rather than a `chan<- cache.UpdateEvents`. This allows us to
do the type conversion without running another goroutine.
2022-05-20 15:47:40 +01:00
Evan Culver 0378372bde
telemetry: remove unused arg (#13161) 2022-05-19 19:17:30 -07:00
funkiestj 6167400b28
api: add the ability to specify a path prefix (#12914)
Specifically meant for when consul is behind a reverse proxy / API gateway

Co-authored-by: Evan Culver <eculver@hashicorp.com>
2022-05-19 16:07:59 -07:00
R.B. Boyer 2e72f44fda
peering: accept replication stream of discovery chain information at the importing side (#13151) 2022-05-19 16:37:52 -05:00
R.B. Boyer c27e186334
test: TestServer_RPC_MetricsIntercept should use a concurrency-safe metrics store (#13157) 2022-05-19 15:39:28 -05:00
DanStough 8513566872 chore(ci): fix rate-limiting for backport-assistant 2022-05-19 16:20:04 -04:00
cskh 364d4f5efe
Retry on bad dogstatsd connection (#13091)
- Introduce a new telemetry configurable parameter retry_failed_connection. User can set the value to true to let consul agent continue its start process on failed connection to datadog server. When set to false, agent will stop on failed start. The default behavior is true.

Co-authored-by: Dan Upton <daniel@floppy.co>
Co-authored-by: Evan Culver <eculver@users.noreply.github.com>
2022-05-19 16:03:46 -04:00
R.B. Boyer c9602bf23e
Merge pull request #13150 from hashicorp/peering-replicate-connect-endpoints
peering: replicate discovery chains information to importing peers
2022-05-19 14:33:30 -05:00
R.B. Boyer 3e4a522882 peering: replicate discovery chains information to importing peers
Treat each exported service as a "discovery chain" and replicate one
synthetic CheckServiceNode for each chain and remote mesh gateway.

The health will be a flattened generated check of the checks for that
mesh gateway node.
2022-05-19 14:21:44 -05:00
R.B. Boyer 5a03536040 prefactor some functions out of the monolithic file 2022-05-19 14:21:29 -05:00
R.B. Boyer 851c8c32b4
test: fix more flakes in the compatibility test (#13145) 2022-05-19 14:05:41 -05:00
R.B. Boyer 1e31dc891a
test: fix incorrect use of t instead of r in retry test (#13146) 2022-05-19 14:00:07 -05:00
DanStough 8b88ad873f chore(ci): squash backport-assistant commits in PRs 2022-05-19 12:35:13 -04:00
Dhia Ayachi 4f17f3ba44
fix 'releases/*.*.x' to 'release/*.*.x' (#13132) 2022-05-19 10:01:57 -04:00
Dmytro a11358ac48
docs: Update service/node identities text on ACL index (#13022)
Modify node and service identities paragraphs on ACL index to better
conform with the style guide.

Co-authored-by: Jeff Boruszak <104028618+boruszak@users.noreply.github.com>
2022-05-18 18:55:35 -07:00
R.B. Boyer 2a4d474d28
test: cleanup and unflake parts of the upgrade compat tests (#13126) 2022-05-18 14:52:26 -05:00
John Murret 0f67ffb984
Updating compatibility matrix for consul-k8s version 0.44.0 (#13111) 2022-05-18 10:05:41 -06:00