Use kubectl's base64decode template function (added in K8s 1.11) to
decode values in Secrets. Removes external call to `base64` utility on
the host system.
Signed-off-by: Jakub Sokołowski <jakub@status.im>
* agent: add failures_before_warning setting
The new setting allows users to specify the number of check failures
that have to happen before a service status us updated to be `warning`.
This allows for more visibility for detected issues without creating
alerts and pinging administrators. Unlike the previous behavior, which
caused the service status to not update until it reached the configured
`failures_before_critical` setting, now Consul updates the Web UI view
with the `warning` state and the output of the service check when
`failures_before_warning` is breached.
The default value of `FailuresBeforeWarning` is the same as the value of
`FailuresBeforeCritical`, which allows for retaining the previous default
behavior of not triggering a warning.
When `FailuresBeforeWarning` is set to a value higher than that of
`FailuresBeforeCritical it has no effect as `FailuresBeforeCritical`
takes precedence.
Resolves: https://github.com/hashicorp/consul/issues/10680
Signed-off-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Jakub Sokołowski <jakub@status.im>
Co-authored-by: Kim Ngo <6362111+findkim@users.noreply.github.com>
Co-authored-by: Melissa Kam <mkam@hashicorp.com>
Co-authored-by: Michael Wilkerson <mwilkerson@hashicorp.com>
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
It can only work if there is a running service instance in the local DC,
so this is a bit misleading, since failover and redirects are typically
used when there is not an instance in the local DC.
Add the list of common Connect CA configuration options to the
provider-specific CA docs.
Previously these options were only documented under the agent
configuration options. This change makes it so that all supported CA
provider configuration options are available from a single location.
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
The current suggests the option expects a string of either "enabled" or "disabled" but this results in an error `'acl.enable_key_list_policy' expected type 'bool', got unconvertible type 'string', value: 'enabled'`. Setting to a boolean value resolves this, also had a quick look at the code (d2b58cd0d6/agent/config/runtime.go (L109)) and it suggests this too
Add a section to the Connect Security page which highlights the risks
of exposing Envoy's administration interface outside of localhost.
Resolves#5692
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Co-authored-by: Kent 'picat' Gruber <kent@hashicorp.com>
Add section for tagged addresses on service definition documentation.
Resolves#6989
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Clarify the function of `-address` flag when instantiating an ingress
gateway.
Resolves#9849
Co-authored-by: trujillo-adam <47586768+trujillo-adam@users.noreply.github.com>
Missed the need to add support for unix domain socket config via
api/command line. This is a variant of the problems described in
it is easy to drop one.
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
This was necessary in older versions of Consul, but was obsoleted by
making Consul add the port number itself when constructing the Envoy
configuration.
* Use CodeTabs for examples in multiple formats.
* Ensure correct language on code fences.
* Use CodeBlockConfig for examples with filenames, or which need
highlighted content.
This commit adds example JSON configs for several config entry
resources were missing examples in this language.
The examples have been updated to use the new CodeTabs resource
instead of the Tab component.
The ServiceChecks parameter was incorrectly documented in e515c9d44 to
state that it accepted a list of string values, when actually the API
requires an array of ServiceCheck objects.
This commit updates the docs for the parameter to correctly reflect
the fields required by the API.
Resolves#10752
Add a note to the docs for the service defaults config entry which
informs users that the service protocol can be configured for all
services using the proxy defaults config entry.
Resolves#8279
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Document the namespace parameter can be specified on HTTP Check,
Connect CA leaf, and Discovery Chain API endpoints.
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
The base64 CLI utility has two different short flag arguments for decode
depending on the platform: -D and -d.
Previously, the docs used the -D flag exclusively with the base64 utility.
Luckily, the long form of the flag is the same across platforms: --decode.
All uses of the base64 -D flag have been replaced with --decode.
Update output for /v1/session/ endpoints to match output post Consul
1.7.0.
Documents new `NodeChecks` and `ServiceChecks` parameters which were
added in that release.
Resolves#7341, resolves#10095
This change adds a new `dns_config.recursor_strategy` option which
controls how Consul queries DNS resolvers listed in the `recursors`
config option. The supported options are `sequential` (default), and
`random`.
Closes#8807
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Priyanka Sengupta <psengupta@flatiron.com>
* add intermediate ca metric routine
* add Gauge config for intermediate cert
* Stop metrics routine when stopping leader
* add changelog entry
* updage changelog
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
* use variables instead of a map
* go imports sort
* Add metrics for primary and secondary ca
* start metrics routine in the right DC
* add telemetry documentation
* update docs
* extract expiry fetching in a func
* merge metrics for primary and secondary into signing ca metric
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Add high level documentation on how to enable ingress controllers in consul on k8s.
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
Co-authored-by: Luke Kysow <1034429+lkysow@users.noreply.github.com>
The docs note an alpha pre-release version on K8s observability. Updating to only reference the GA 1.10.0 version of Consul for observability on k8s and metrics merging.
The main branch is being renamed from master->main. This commit should
update all references to the main branch to the new name.
Co-Authored-By: Mike Morris <mikemorris@users.noreply.github.com>
* docs: Add info on using cloud auto-join with Network Segments
Resolves hashicorp/go-discover#57
* Add note about joining network segments
Specifically call out that agents can be configured to join a network
segment by either specifying the Serf LAN port in the join address,
changing the agent's default Serf LAN port by configuring
`ports.serf_lan`, or specifying the port in the `-serf-lan-port`
command line flag.
Resolves#9972
* docs: Remove Helm 2 mention in Consul K8s install and uninstall
Helm 2 is no longer supported via Consul K8s. Helm 3 is now the supported version for Consul K8s.
The query metrics are actually reported for all read queries, not only
ones that use a MinIndex to block for updates.
Also clarify the raft.apply metric is only on the leader.
* docs: Removal of Consul vs ZooKeeper
Although Consul does have a KV, we are not positioning Consul as a first class KV store versus other alternatives such as etcd or Zookeeper. Will remove this since this has not been updated with further analysis since this content was created.
* Removing from Zookeeper analysis Navbar
* Removing Zookeeper analysis from redirects
* docs/nia: Add section on upgrading Terraform in CTS
* docs/nia: Add service filter configuration, deprecate tag
* docs/nia: Add version to deprecated note, use path to reference
* docs/nia: catalog-services condition
Co-authored-by: Melissa Kam <mkam@hashicorp.com>
Co-authored-by: Melissa Kam <3768460+mkam@users.noreply.github.com>
CatalogDestinationsOnly is a passthrough that would enable dialing
addresses outside of Consul's catalog. However, when this flag is set to
true only _connect_ endpoints for services can be dialed.
This flag is being renamed to signal that non-Connect endpoints can't be
dialed by transparent proxies when the value is set to true.
Previously if you were to follow these docs and register two external
services, you would set the Address field on the node. The second
registered service would change the address of the node for the first
service.
Now the docs explain the address key and how to register more than one
external service.
* debug: remove the CLI check for debug_enabled
The API allows collecting profiles even debug_enabled=false as long as
ACLs are enabled. Remove this check from the CLI so that users do not
need to set debug_enabled=true for no reason.
Also:
- fix the API client to return errors on non-200 status codes for debug
endpoints
- improve the failure messages when pprof data can not be collected
Co-Authored-By: Dhia Ayachi <dhia@hashicorp.com>
* remove parallel test runs
parallel runs create a race condition that fail the debug tests
* snapshot the timestamp at the beginning of the capture
- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist
* Revert "snapshot the timestamp at the beginning of the capture"
This reverts commit c2d03346
* Refactor captureDynamic to extract capture logic for each item in a different func
* snapshot the timestamp at the beginning of the capture
- timestamp used to create the capture sub folder is snapshot only at the beginning of the capture and reused for subsequent captures
- capture append to the file if it already exist
* Revert "snapshot the timestamp at the beginning of the capture"
This reverts commit c2d03346
* Refactor captureDynamic to extract capture logic for each item in a different func
* extract wait group outside the go routine to avoid a race condition
* capture pprof in a separate go routine
* perform a single capture for pprof data for the whole duration
* add missing vendor dependency
* add a change log and fix documentation to reflect the change
* create function for timestamp dir creation and simplify error handling
* use error groups and ticker to simplify interval capture loop
* Logs, profile and traces are captured for the full duration. Metrics, Heap and Go routines are captured every interval
* refactor Logs capture routine and add log capture specific test
* improve error reporting when log test fail
* change test duration to 1s
* make time parsing in log line more robust
* refactor log time format in a const
* test on log line empty the earliest possible and return
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
* rename function to captureShortLived
* more specific changelog
Co-authored-by: Paul Banks <banks@banksco.de>
* update documentation to reflect current implementation
* add test for behavior when invalid param is passed to the command
* fix argument line in test
* a more detailed description of the new behaviour
Co-authored-by: Paul Banks <banks@banksco.de>
* print success right after the capture is done
* remove an unnecessary error check
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
* upgraded github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57 => v0.0.0-20210601050228-01bbb1931b22
Co-authored-by: Daniel Nephin <dnephin@hashicorp.com>
Co-authored-by: Freddy <freddygv@users.noreply.github.com>
Co-authored-by: Paul Banks <banks@banksco.de>
* Docs for Unix Domain Sockets
There are a number of cases where a user might wish to either 1)
expose a service through a Unix Domain Socket in the filesystem
('downstream') or 2) connect to an upstream service by a local unix
domain socket (upstream).
As of Consul (1.10-beta2) we've added new syntax and support to configure
the Envoy proxy to support this
To connect to a service via local Unix Domain Socket instead of a
port, add local_bind_socket_path and optionally local_bind_socket_mode
to the upstream config for a service:
upstreams = [
{
destination_name = "service-1"
local_bind_socket_path = "/tmp/socket_service_1"
local_bind_socket_mode = "0700"
...
}
...
]
This will cause Envoy to create a socket with the path and mode
provided, and connect that to service-1
The mode field is optional, and if omitted will use the default mode
for Envoy. This is not applicable for abstract sockets. See
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/address.proto#envoy-v3-api-msg-config-core-v3-pipe
for details
NOTE: These options conflict the local_bind_socket_port and
local_bind_socket_address options. We can bind to an port or we can
bind to a socket, but not both.
To expose a service listening on a Unix Domain socket to the service
mesh use either the 'socket_path' field in the service definition or the
'local_service_socket_path' field in the proxy definition. These
fields are analogous to the 'port' and 'service_port' fields in their
respective locations.
services {
name = "service-2"
socket_path = "/tmp/socket_service_2"
...
}
OR
proxy {
local_service_socket_path = "/tmp/socket_service_2"
...
}
There is no mode field since the service is expected to create the
socket it is listening on, not the Envoy proxy.
Again, the socket_path and local_service_socket_path fields conflict
with address/port and local_service_address/local_service_port
configuration entries.
Set up a simple service mesh with dummy services:
socat -d UNIX-LISTEN:/tmp/downstream.sock,fork UNIX-CONNECT:/tmp/upstream.sock
socat -v tcp-l:4444,fork exec:/bin/cat
services {
name = "sock_forwarder"
id = "sock_forwarder.1"
socket_path = "/tmp/downstream.sock"
connect {
sidecar_service {
proxy {
upstreams = [
{
destination_name = "echo-service"
local_bind_socket_path = "/tmp/upstream.sock"
config {
passive_health_check {
interval = "10s"
max_failures = 42
}
}
}
]
}
}
}
}
services {
name = "echo-service"
port = 4444
connect = { sidecar_service {} }
Kind = "ingress-gateway"
Name = "ingress-service"
Listeners = [
{
Port = 8080
Protocol = "tcp"
Services = [
{
Name = "sock_forwarder"
}
]
}
]
consul agent -dev -enable-script-checks -config-dir=./consul.d
consul connect envoy -sidecar-for sock_forwarder.1
consul connect envoy -sidecar-for echo-service -admin-bind localhost:19001
consul config write ingress-gateway.hcl
consul connect envoy -gateway=ingress -register -service ingress-service -address '{{ GetInterfaceIP "eth0" }}:8888' -admin-bind localhost:19002
netcat 127.0.0.1 4444
netcat 127.0.0.1 8080
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
* fixup Unix capitalization
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
* Update website/content/docs/connect/registration/service-registration.mdx
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
* Provide examples in hcl and json
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
* Apply suggestions from code review
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
* One more fixup for docs
Signed-off-by: Mark Anderson <manderson@hashicorp.com>
Co-authored-by: Blake Covarrubias <blake@covarrubi.as>
This PR adds cluster members to the metrics API. The number of members per
segment are reported as well as the total number of members.
Tested by running a multi-node cluster locally and ensuring the numbers were
correct. Also added unit test coverage to add the new expected gauges to
existing test cases.
Adds more clear indicators that the collections on the learn.hashicorp.com sites have specific instructions for single node deployments.
Co-Authored by: soonoo <qpseh2m7@gmail.com>
* Update glossary.mdx
1. Update header to the first section to "Consul Vocabulary" since these are the terms used in the context of Consul conversations.
2. Kept the header "Consul Glossary" since these are the terms useful for practitioners in the consul space.
3. Removed interlinking to terms on the same page.
Co-authored-by: Hans Hasselberg <me@hans.io>
Co-authored-by: Swarna Podila <swarnap@users.noreply.github.com>
Update register check documentation clarify that Id returns as CheckId in the response
Co-Authored-By: Shaker Islam <shaqq@users.noreply.github.com>
Co-authored-by: Shaker Islam <shaqq@users.noreply.github.com>
Per Consul PM, kubeconfig is not required for manual join. I believe this should be clarified in the docs as the current wording refers to the auto join steps above which state kubeconfig is required.
* WIP reloadable raft config
* Pre-define new raft gauges
* Update go-metrics to change gauge reset behaviour
* Update raft to pull in new metric and reloadable config
* Add snapshot persistance timing and installSnapshot to our 'protected' list as they can be infrequent but are important
* Update telemetry docs
* Update config and telemetry docs
* Add note to oldestLogAge on when it is visible
* Add changelog entry
* Update website/content/docs/agent/options.mdx
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
Co-authored-by: Matt Keeler <mkeeler@users.noreply.github.com>
Initially we were loading every potential upstream address into Envoy
and then routing traffic to the logical upstream service. The downside
of this behavior is that traffic meant to go to a specific instance
would be load balanced across ALL instances.
Traffic to specific instance IPs should be forwarded to the original
destination and if it's a destination in the mesh then we should ensure
the appropriate certificates are used.
This PR makes transparent proxying a Kubernetes-only feature for now
since support for other environments requires generating virtual IPs,
and Consul does not do that at the moment.
The only thing that needed fixing up pertained to this section of the 1.18.x release notes:
> grpc_stats: the default value for stats_for_all_methods is switched from true to false, in order to avoid possible memory exhaustion due to an untrusted downstream sending a large number of unique method names. The previous default value was deprecated in version 1.14.0. This only changes the behavior when the value is not set. The previous behavior can be used by setting the value to true. This behavior change by be overridden by setting runtime feature envoy.deprecated_features.grpc_stats_filter_enable_stats_for_all_methods_by_default.
For now to maintain status-quo I'm explicitly setting `stats_for_all_methods=true` in all versions to avoid relying upon the default.
Additionally the naming of the emitted metrics for these gRPC requests changed slightly so the integration test assertions for `case-grpc` needed adjusting.
* Update website/content/docs/discovery/services.mdx with address field behavior.
Co-authored-by: Jono Sosulska <42216911+jsosulska@users.noreply.github.com>
Co-authored-by: Jono Sosulska <42216911+jsosulska@users.noreply.github.com>
* Use proxy outbound port from TransparentProxyConfig if provided
* If -proxy-id is provided to the redirect-traffic command, exclude any listener ports
from inbound traffic redirection. This includes envoy_prometheus_bind_addr,
envoy_stats_bind_addr, and the ListenerPort from the Expose configuration.
* Allow users to provide additional inbound and outbound ports, outbound CIDRs
and additional user IDs to be excluded from traffic redirection.
This affects both the traffic-redirect command and the iptables SDK package.
This config entry is being renamed primarily because in k8s the name
cluster could be confusing given that the config entry applies across
federated datacenters.
Additionally, this config entry will only apply to Consul as a service
mesh, so the more generic "cluster" name is not needed.
* CLI: Add support for reading internal raft snapshots to snapshot inspect
* Add snapshot inspect test for raw state files
* Add changelog entry
* Update .changelog/10089.txt
* add http2 ping checks
* fix test issue
* add h2ping check to config resources
* add new test and docs for h2ping
* fix grammatical inconsistency in H2PING documentation
* resolve rebase conflicts, add test for h2ping tls verification failure
* api documentation for h2ping
* update test config data with H2PING
* add H2PING to protocol buffers and update changelog
* fix typo in changelog entry
* Add new consul connect redirect-traffic command for applying traffic redirection rules when Transparent Proxy is enabled.
* Add new iptables package for applying traffic redirection rules with iptables.
* website: migrate to new nav-data format
* website: clean up unused intro content
* website: remove deprecated sidebar_title from frontmatter
* website: add react-content to fix global style import issue
The Agent token policy when using rexec should have `write` on "_rexec"
key prefix. Updated the exec command documentation to explicitly state
this requirement.
* Fixes #2379-Improve interval explanation in the telemetry doc
* Fixes #4734-Update consul memory metrics
* Fixes #4836-Removed node.deregistration as that isn't in state.go
* Fixes#8986 partially-Trim redundant language
* Fixes #9087-Adds helpful details to telemetry on autopilot
* Fixes #9274-Addresses NaN output in autopilot
Some TLS servers require SNI, but the Golang HTTP client doesn't
include it in the ClientHello when connecting to an IP address. This
change adds a new TLSServerName field to health check definitions to
optionally set it. This fixes#9473.
Allows setting -prometheus-backend-port to configure the cluster
envoy_prometheus_bind_addr points to.
Allows setting -prometheus-scrape-path to configure which path
envoy_prometheus_bind_addr exposes metrics on.
-prometheus-backend-port is used by the consul-k8s metrics merging feature, to
configure envoy_prometheus_bind_addr to point to the merged metrics
endpoint that combines Envoy and service metrics so that one set of
annotations on a Pod can scrape metrics from the service and it's Envoy
sidecar.
-prometheus-scrape-path is used to allow configurability of the path
where prometheus metrics are exposed on envoy_prometheus_bind_addr.
The Server certificates used for Federation require the node name in the form of `<node>.server.<dc>.<domain>`. Not having this would through `bad tls certificate` error.
* Fixed cert create command
* Added note to create a wildcard cert (like the ones on Kubernetes)
* Fixed numbering
nia/docs 0.1.0-beta
Co-authored-by: Kent 'picat' Gruber <kent@hashicorp.com>
Co-authored-by: Lorna Song <lorna@hashicorp.com>
Co-authored-by: John Eikenberry <jae@zhar.net>
Note that this does NOT upgrade to xDS v3. That will come in a future PR.
Additionally:
- Ignored staticcheck warnings about how github.com/golang/protobuf is deprecated.
- Shuffled some agent/xds imports in advance of a later xDS v3 upgrade.
- Remove support for envoy 1.13.x but don't add in 1.17.x yet. We have to wait until the xDS v3 support is added in a follow-up PR.
Fixes#8425
Currently when data is imported via `consul kv import` it overwrites
keys under the root key. Since `consul kv export` can retrieve data for
the given prefix, i.e. part of the KV tree, importing it under root may
be not what users want.
To mirror prefix behavior from export this PR adds prefix feature to the
import command that adds prefix to all keys that are imported.
* Document how users can migrate to CRDs.
* Update documentation for federation with new `ProxyDefaults`
requirement.
* Ensure `controller.enabled: true` is set in our example configs.
* Remove `connect-service-protocol` annotation docs.
* Add docs for MIME sniffing on metrics endpoint.
This was added in 1.7.2 last year but I realised we don't document it so it's kinda surprising Prometheus "just works" now.
* Update website/content/api-docs/agent/index.mdx
This allows setting ForceWithoutCrossSigning when reconfiguring the CA
for any provider, in order to forcibly move to a new root in cases where
the old provider isn't reachable or able to cross-sign for whatever
reason.
Luke Kysow
Evan Culver
Jared Kirschner
Jared Kirschner
Paul Banks
Chris S. Kim
David Yu
Paul Banks
Blake Covarrubias
trujillo-adam
trujillo-adam
Daniel Nephin
CJ
Connor
Ashwin Venkatesh
Evan Culver
Iryna Shustava
Paul Glass
Kyle MacDonald
mrspanishviking
Peter M
Amos Shapira
lornasong
Freddy
Dustin Clifford
sridhar
Nitya Dhanushkodi
Olatunde Alex-Oni
Mark Anderson
Chris Piraino
joshwolfer
Matt Explosion
Sabeen Syed
Fabio Lichinchi
Melissa Kam
Melissa Kam
Curt Marker
Dhia Ayachi
Kyle Schochenmaier
Jan Berktold
Mike Ajala
Ludovic Ortega
Kim Ngo
Matt Keeler
R.B. Boyer
Christopher Lai
Nick Wales
allisaurus
Paul Ewing
Matt Keeler
Stanko
Jono Sosulska
mrspanishviking
Joel Watson
Andy Assareh
Seth Hoenig
Frederic Hemberger
Luigi Tagliamonte
Derek Strickland
Kent 'picat' Gruber
ketzacoatl
Tara Tufano
Zachary Shilton
eddie-rowe
Mike Green
danielehc
Ranjandas
woz5999
Christoph Puhl
Christopher Broglie
Mike Wickett
Preetha
Kyle Havlovitz
Daniel Mabbett
Robert Kuska
R.B. Boyer
Kim Ngo
kaitlincarter-hc
Alex Dzyoba
Mike Morris
Jeff Escalante
Chris Boulton
Mustafa Sadedil
markblackman
Mark Lewis