* debug: remove the CLI check for debug_enabled
The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.
Also:
- fix the API client to return errors on non-200 status codes for debug
endpoints
- improve the failure messages when pprof data can not be collected
Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>
* remove parallel test runs
parallel runs create a race condition that fail the debug tests
* snapshot the timestamp at the beginning of the capture
- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist
* Revert "snapshot the timestamp at the beginning of the capture"
This reverts commit c2d03346
* Refactor captureDynamic to extract capture logic for each item in a different func
* snapshot the timestamp at the beginning of the capture
- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist
* Revert "snapshot the timestamp at the beginning of the capture"
This reverts commit c2d03346
* Refactor captureDynamic to extract capture logic for each item in a different func
* extract wait group outside the go routine to avoid a race condition
* capture pprof in a separate go routine
* perform a single capture for pprof data for the whole duration
* add missing vendor dependency
* add a change log and fix documentation to reflect the change
* create function for timestamp dir creation and simplify error handling
* use error groups and ticker to simplify interval capture loop
* Logs, profile and traces are captured for the full duration. Metrics, Heap and Go routines are captured every interval
* refactor Logs capture routine and add log capture specific test
* improve error reporting when log test fail
* change test duration to 1s
* make time parsing in log line more robust
* refactor log time format in a const
* test on log line empty the earliest possible and return
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
* rename function to captureShortLived
* more specific changelog
Co-authored-by: Paul Banks <banks@banksco.de>
* update documentation to reflect current implementation
* add test for behavior when invalid param is passed to the command
* fix argument line in test
* a more detailed description of the new behaviour
Co-authored-by: Paul Banks <banks@banksco.de>
* print success right after the capture is done
* remove an unnecessary error check
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
* upgraded github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57 => v0.0.0-20210601050228-01bbb1931b22
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: Paul Banks <banks@banksco.de>
* Docs for Unix Domain Sockets
There are a number of cases where a user might wish to either 1)
expose a service through a Unix Domain Socket in the filesystem
('downstream') or 2) connect to an upstream service by a local unix
domain socket (upstream).
As of Consul (1.10-beta2) we've added new syntax and support to configure
the Envoy proxy to support this
To connect to a service via local Unix Domain Socket instead of a
port, add local_bind_socket_path and optionally local_bind_socket_mode
to the upstream config for a service:
upstreams = [
{
destination_name = "service-1"
local_bind_socket_path = "/tmp/socket_service_1"
local_bind_socket_mode = "0700"
...
}
...
]
This will cause Envoy to create a socket with the path and mode
provided, and connect that to service-1
The mode field is optional, and if omitted will use the default mode
for Envoy. This is not applicable for abstract sockets. See
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-pipe
for details
NOTE: These options conflict the local_bind_socket_port and
local_bind_socket_address options. We can bind to an port or we can
bind to a socket, but not both.
To expose a service listening on a Unix Domain socket to the service
mesh use either the 'socket_path' field in the service definition or the
'local_service_socket_path' field in the proxy definition. These
fields are analogous to the 'port' and 'service_port' fields in their
respective locations.
services {
name = "service-2"
socket_path = "/tmp/socket_service_2"
...
}
OR
proxy {
local_service_socket_path = "/tmp/socket_service_2"
...
}
There is no mode field since the service is expected to create the
socket it is listening on, not the Envoy proxy.
Again, the socket_path and local_service_socket_path fields conflict
with address/port and local_service_address/local_service_port
configuration entries.
Set up a simple service mesh with dummy services:
socat -d UNIX-LISTEN:/tmp/downstream.sock,fork UNIX-CONNECT:/tmp/upstream.sock
socat -v tcp-l:4444,fork exec:/bin/cat
services {
name = "sock_forwarder"
id = "sock_forwarder.1"
socket_path = "/tmp/downstream.sock"
connect {
sidecar_service {
proxy {
upstreams = [
{
destination_name = "echo-service"
local_bind_socket_path = "/tmp/upstream.sock"
config {
passive_health_check {
interval = "10s"
max_failures = 42
}
}
}
]
}
}
}
}
services {
name = "echo-service"
port = 4444
connect = { sidecar_service {} }
Kind = "ingress-gateway"
Name = "ingress-service"
Listeners = [
{
Port = 8080
Protocol = "tcp"
Services = [
{
Name = "sock_forwarder"
}
]
}
]
consul agent -dev -enable-script-checks -config-dir=./consul.d
consul connect envoy -sidecar-for sock_forwarder.1
consul connect envoy -sidecar-for echo-service -admin-bind localhost:19001
consul config write ingress-gateway.hcl
consul connect envoy -gateway=ingress -register -service ingress-service -address '{{ GetInterfaceIP "eth0" }}:8888' -admin-bind localhost:19002
netcat 127.0.0.1 4444
netcat 127.0.0.1 8080
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
* fixup Unix capitalization
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
* Update website/content/docs/connect/registration/service-registration.mdx
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
* Provide examples in hcl and json
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
* Apply suggestions from code review
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
* One more fixup for docs
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
This PR adds cluster members to the metrics API. The number of members per
segment are reported as well as the total number of members.
Tested by running a multi-node cluster locally and ensuring the numbers were
correct. Also added unit test coverage to add the new expected gauges to
existing test cases.
Adds more clear indicators that the collections on the learn.hashicorp.com sites have specific instructions for single node deployments.
Co-Authored by: soonoo <qpseh2m7@gmail.com>
* Update glossary.mdx
1. Update header to the first section to "Consul Vocabulary" since these are the terms used in the context of Consul conversations.
2. Kept the header "Consul Glossary" since these are the terms useful for practitioners in the consul space.
3. Removed interlinking to terms on the same page.
Co-authored-by: Hans Hasselberg <me@hans.io>
Co-authored-by: Swarna Podila <swarnap@users.noreply.github.com>
Update register check documentation clarify that Id returns as CheckId in the response
Co-Authored-By: Shaker Islam <shaqq@users.noreply.github.com>
Co-authored-by: Shaker Islam <shaqq@users.noreply.github.com>
Plug in callouts content
Most styles for the callout blade
Re-export Consul Stack image
Update icons
Misc content tweaks
Optimize mobile for the consul stack icon
Better alignment
Make full CTA box clickable
Per Consul PM, kubeconfig is not required for manual join. I believe this should be clarified in the docs as the current wording refers to the auto join steps above which state kubeconfig is required.
* website: bump to docs-page prerelease with hidden page support
* website: replace hidden pages hot fix for /docs, /api-docs, and /commmands
* website: remove unused files for old hidden pages hotfix
* website: bump to stable docs-page, w next-mdx-remote bump
* website: bump to latest markdown-page
* WIP reloadable raft config
* Pre-define new raft gauges
* Update go-metrics to change gauge reset behaviour
* Update raft to pull in new metric and reloadable config
* Add snapshot persistance timing and installSnapshot to our 'protected' list as they can be infrequent but are important
* Update telemetry docs
* Update config and telemetry docs
* Add note to oldestLogAge on when it is visible
* Add changelog entry
* Update website/content/docs/agent/options.mdx
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
Initially we were loading every potential upstream address into Envoy
and then routing traffic to the logical upstream service. The downside
of this behavior is that traffic meant to go to a specific instance
would be load balanced across ALL instances.
Traffic to specific instance IPs should be forwarded to the original
destination and if it's a destination in the mesh then we should ensure
the appropriate certificates are used.
This PR makes transparent proxying a Kubernetes-only feature for now
since support for other environments requires generating virtual IPs,
and Consul does not do that at the moment.
The only thing that needed fixing up pertained to this section of the 1.18.x release notes:
> grpc_stats: the default value for stats_for_all_methods is switched from true to false, in order to avoid possible memory exhaustion due to an untrusted downstream sending a large number of unique method names. The previous default value was deprecated in version 1.14.0. This only changes the behavior when the value is not set. The previous behavior can be used by setting the value to true. This behavior change by be overridden by setting runtime feature envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default.
For now to maintain status-quo I'm explicitly setting `stats_for_all_methods=true` in all versions to avoid relying upon the default.
Additionally the naming of the emitted metrics for these gRPC requests changed slightly so the integration test assertions for `case-grpc` needed adjusting.
* Update website/content/docs/discovery/services.mdx with address field behavior.
Co-authored-by: Jono Sosulska <42216911+jsosulska@users.noreply.github.com>
Co-authored-by: Jono Sosulska <42216911+jsosulska@users.noreply.github.com>
* Use proxy outbound port from TransparentProxyConfig if provided
* If -proxy-id is provided to the redirect-traffic command, exclude any listener ports
from inbound traffic redirection. This includes envoy_prometheus_bind_addr,
envoy_stats_bind_addr, and the ListenerPort from the Expose configuration.
* Allow users to provide additional inbound and outbound ports, outbound CIDRs
and additional user IDs to be excluded from traffic redirection.
This affects both the traffic-redirect command and the iptables SDK package.
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.
Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
* CLI: Add support for reading internal raft snapshots to snapshot inspect
* Add snapshot inspect test for raw state files
* Add changelog entry
* Update .changelog/10089.txt
* website: add back unlinked pages to match previous state
* website: add unlinked content check
* website: add hidden nav-data to unlinked content check
* add http2 ping checks
* fix test issue
* add h2ping check to config resources
* add new test and docs for h2ping
* fix grammatical inconsistency in H2PING documentation
* resolve rebase conflicts, add test for h2ping tls verification failure
* api documentation for h2ping
* update test config data with H2PING
* add H2PING to protocol buffers and update changelog
* fix typo in changelog entry
* Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled.
* Add new iptables package for applying traffic redirection rules with iptables.
* website: migrate to new nav-data format
* website: clean up unused intro content
* website: remove deprecated sidebar_title from frontmatter
* website: add react-content to fix global style import issue
The Agent token policy when using rexec should have `write` on "_rexec"
key prefix. Updated the exec command documentation to explicitly state
this requirement.
* Fixes #2379-Improve interval explanation in the telemetry doc
* Fixes #4734-Update consul memory metrics
* Fixes #4836-Removed node.deregistration as that isn't in state.go
* Fixes#8986 partially-Trim redundant language
* Fixes #9087-Adds helpful details to telemetry on autopilot
* Fixes #9274-Addresses NaN output in autopilot
Some TLS servers require SNI, but the Golang HTTP client doesn't
include it in the ClientHello when connecting to an IP address. This
change adds a new TLSServerName field to health check definitions to
optionally set it. This fixes#9473.
Allows setting -prometheus-backend-port to configure the cluster
envoy_prometheus_bind_addr points to.
Allows setting -prometheus-scrape-path to configure which path
envoy_prometheus_bind_addr exposes metrics on.
-prometheus-backend-port is used by the consul-k8s metrics merging feature, to
configure envoy_prometheus_bind_addr to point to the merged metrics
endpoint that combines Envoy and service metrics so that one set of
annotations on a Pod can scrape metrics from the service and it's Envoy
sidecar.
-prometheus-scrape-path is used to allow configurability of the path
where prometheus metrics are exposed on envoy_prometheus_bind_addr.
The Server certificates used for Federation require the node name in the form of `<node>.server.<dc>.<domain>`. Not having this would through `bad tls certificate` error.
* Fixed cert create command
* Added note to create a wildcard cert (like the ones on Kubernetes)
* Fixed numbering
nia/docs 0.1.0-beta
Co-authored-by: Kent 'picat' Gruber <kent@hashicorp.com>
Co-authored-by: Lorna Song <lorna@hashicorp.com>
Co-authored-by: John Eikenberry <jae@zhar.net>
Note that this does NOT upgrade to xDS v3. That will come in a future PR.
Additionally:
- Ignored staticcheck warnings about how github.com/golang/protobuf is deprecated.
- Shuffled some agent/xds imports in advance of a later xDS v3 upgrade.
- Remove support for envoy 1.13.x but don't add in 1.17.x yet. We have to wait until the xDS v3 support is added in a follow-up PR.
Fixes#8425
Currently when data is imported via `consul kv import` it overwrites
keys under the root key. Since `consul kv export` can retrieve data for
the given prefix, i.e. part of the KV tree, importing it under root may
be not what users want.
To mirror prefix behavior from export this PR adds prefix feature to the
import command that adds prefix to all keys that are imported.
* Document how users can migrate to CRDs.
* Update documentation for federation with new `ProxyDefaults`
requirement.
* Ensure `controller.enabled: true` is set in our example configs.
* Remove `connect-service-protocol` annotation docs.
* Add docs for MIME sniffing on metrics endpoint.
This was added in 1.7.2 last year but I realised we don't document it so it's kinda surprising Prometheus "just works" now.
* Update website/content/api-docs/agent/index.mdx
This allows setting ForceWithoutCrossSigning when reconfiguring the CA
for any provider, in order to forcibly move to a new root in cases where
the old provider isn't reachable or able to cross-sign for whatever
reason.
This PR is based on the previous work by @snuggie12 in PR #6825. It adds the command consul intention list to list all available intentions. The list functionality for intentions seems a bit overdue as it's just very handy. The web UI cannot list intentions outside of the default namespace, and using the API is sometimes not the friendliest option. ;)
I cherry picked snuggie12's commits who did most of the heavy lifting (thanks again @snuggie12 for your great work!). The changes in the original commit mostly still worked on the current HEAD. On top of that I added support for namespaces and fixed the docs as they are managed differently today. Also the requested changes related to the "Connect" references in the original PRs have been addressed.
Fixes#5652
Co-authored-by: Matt Hoey <mhoey05@jcu.edu>
CTS running with default configuration will communicate over http
unless the Consul client is configured with TLS. Having the example
set the scheme to https is misleading and will result in an error:
"http: server gave HTTP response to HTTPS client"
Hi everyone,
I'm developing an open source CLI tool for easily editing Consul KV values from your favorite text editor. I've added to Client Libraries & SDKs page on website.
Repository: https://github.com/sadedil/kvit
Kind regards,
Mustafa
* Adds a new react component `ConfigEntryReference` that allows us to
document config entries for both HCL and Kubernetes CRDs.
* Rewrites config entry docs to use new component
* Add CRD examples alongside HCL examples
* Removes examples from Kube CRD doc because the examples are now in the
main CRD docs
* update example for tag_name
In the example for azure the "tag_name" field is wrong cause say "tag_key" and not "tag_name"
* change tag_key by tag_name in the description of azure
* reorganize for clarity and update for value syntax
* fix quotes around value
* Apply suggestions from code review
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
* Apply suggestions from code review
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
* create consul version metric with version label
* agent/agent.go: add pre-release Version as well as label
Co-Authored-By: Radha13 <kumari.radha3@gmail.com>
* verion and pre-release version labels.
* hyphen/- breaks prometheus
* Add Prometheus gauge defintion for version metric
* Add new metric to telemetry docs
Co-authored-by: Radha Kumari <kumari.radha3@gmail.com>
Co-authored-by: Aestek <thib.gilles@gmail.com>
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
- Fix anchors for client.extraEnvironmentVars and
server.extraEnvironmentVars.
- Change extraEnvironmentVars data type to `map`.
- Fix external link to kubernetes.io under
connectInject.namespaceSelector.
A vulnerability was identified in Consul and Consul Enterprise (“Consul”) such that operators with `operator:read` ACL permissions are able to read the Consul Connect CA configuration when explicitly configured with the `/v1/connect/ca/configuration` endpoint, including the private key. This allows the user to effectively privilege escalate by enabling the ability to mint certificates for any Consul Connect services. This would potentially allow them to masquerade (receive/send traffic) as any service in the mesh.
--
This PR increases the permissions required to read the Connect CA's private key when it was configured via the `/connect/ca/configuration` endpoint. They are now `operator:write`.
Header is: X-Consul-Default-ACL-Policy=<allow|deny>
This is of particular utility when fetching matching intentions, as the
fallthrough for a request that doesn't match any intentions is to
enforce using the default acl policy.
It's better to avoid the ambiguous Vault statement that was not clarified and drop the loaded "roles" term in favor of "capabilities" since the ACL system is described as capability-based in previous ACL documentation.
Consul's Connect CA documentation mentions future releases will
support a pluggable CA system. This sentence has existed in the docs
for over two years, however there are currently no plans to develop
this feature on the near-term roadmap.
This commit removes this sentence to avoid giving the impression that
this feature will be available in an upcoming release.
* Update to CRD docs
* Update website/pages/docs/k8s/crds.mdx
* Modify proxy default and service default protocols
Carry over from previous PR that I forgot to submit a review/suggestion to, TCP and HTTP are not valid protocols for Proxy Defaults and Service Defaults
kubectl apply -f sdefault.yml
Error from server: error when creating "sdefault.yml": admission webhook "mutate-servicedefaults.consul.hashicorp.com" denied the request: servicedefaults.consul.hashicorp.com "your-service-name" is invalid: spec.expose.paths[0].protocol: Invalid value: "tcp": must be one of "http", "http2"
kubectl apply -f sdefault.yml
Error from server: error when creating "sdefault.yml": admission webhook "mutate-servicedefaults.consul.hashicorp.com" denied the request: servicedefaults.consul.hashicorp.com "your-service-name" is invalid: spec.expose.paths[0].protocol: Invalid value: "tcp": must be one of "http", "http2"
Co-authored-by: David Yu <dyu@hashicorp.com>
Peter M
Daniel Nephin
Mike Morris
Brandon Romano
Dhia Ayachi
allisaurus
Mark Anderson
Jeff Escalante
Matt Keeler
Paul Ewing
Matt Keeler
Jimmy Merritello
Kendall Strautman
Blake Covarrubias
Stanko
Bryce Kalow
David Yu
Luke Kysow
Jono Sosulska
Karl Cardenas
Sabeen Syed
Paul Banks
Kendall Strautman
R.B. Boyer
Christopher Poenaru
mrspanishviking
Brandon Romano
Kyle MacDonald
Joel Watson
Kim Ngo
Andy Assareh
Seth Hoenig
Zachary Shilton
Freddy
Frederic Hemberger
Luigi Tagliamonte
Iryna Shustava
Derek Strickland
Kent 'picat' Gruber
Mike Wickett
ketzacoatl
Mike Wickett
Tara Tufano
eddie-rowe
Noel Quiles
Mike Green
lornasong
danielehc
Ranjandas
Nitya Dhanushkodi
woz5999
Christoph Puhl
Christopher Broglie
Preetha
Kyle Havlovitz
Daniel Mabbett
Robert Kuska
R.B. Boyer
Kim Ngo
kaitlincarter-hc
Alex Dzyoba
Ashwin Venkatesh
Kyle MacDonald
Chris Boulton
Mustafa Sadedil
markblackman
freddygv
Mark Lewis
Chulki Lee
Michael Hofer
Michael Montgomery
Tenpo
Maksym
Kenia
Max G
Rob Taylor
Hans Hasselberg
Kit Patella
Kyle Schochenmaier
Sean Ellefson
Alvin Huang
s-christoff
Kevin Pruett
Kevin Pruett
James Light
Jasmine W
Jasmine W
Lorna Song
pcmccarron
Ricardo Oliveira