consul/docs/v2-architecture/controller-architecture/guide.md

# Resource and Controller Developer Guide

This is a whistle-stop tour through adding a new resource type and controller to
Consul 🚂

## Resource Schema

Adding a new resource type begins with defining the object schema as a protobuf
message, in the appropriate package under [`proto-public`](../../../proto-public).

```shell
$ mkdir proto-public/pbfoo/v1alpha1
```

```proto
// proto-public/pbfoo/v1alpha1/foo.proto
syntax = "proto3";

import "pbresource/resource.proto";
import "pbresource/annotations.proto";

package hashicorp.consul.foo.v1alpha1;

message Bar {
  option (hashicorp.consul.resource.spec) = {scope: SCOPE_NAMESPACE};
  
  string baz = 1;
  hashicorp.consul.resource.ID qux = 2;
}
```

```shell
$ make proto
```

Next, we must add our resource type to the registry. At this point, it's useful
to add a package (e.g. under [`internal`](../../../internal)) to contain the logic
associated with this resource type.

The convention is to have this package export variables for its type identifiers
along with a method for registering its types:

```Go
// internal/foo/types.go
package foo

import (
	"github.com/hashicorp/consul/internal/resource"
	pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:  pbv1alpha1.BarType, 
		Scope: resource.ScopePartition,
		Proto: &pbv1alpha1.Bar{},
	})
}
```
Note that Scope reference the scope of the new resource, `resource.ScopePartition` 
mean that resource will be at the partition level and have no namespace, while `resource.ScopeNamespace` mean it will have both a namespace 
and a partition.

Update the `NewTypeRegistry` method in [`type_registry.go`] to call your
package's type registration method:

[`type_registry.go`]: ../../../agent/consul/type_registry.go

```Go
import (
	// …
	"github.com/hashicorp/consul/internal/foo"
	// …
)

func NewTypeRegistry() resource.Registry {
	// …
    foo.RegisterTypes(registry)
	// …
}
```

That should be all you need to start using your new resource type. Test it out
by starting an agent in dev mode:

```shell
$ make dev
$ consul agent -dev
```

You can now use [grpcurl](https://github.com/fullstorydev/grpcurl) to interact
with the [resource service](../../../proto-public/pbresource/resource.proto):

```shell
$ grpcurl -d @ \
  -plaintext \
  -protoset pkg/consul.protoset \
  127.0.0.1:8502 \
  hashicorp.consul.resource.ResourceService.Write \
<<EOF
  {
    "resource": {
      "id": {
        "type": {
          "group": "foo",
          "group_version": "v1alpha1",
          "kind": "bar"
        },
        "tenancy": {
          "partition": "default",
          "peer_name": "local",
          "namespace": "default"
        }
      },
      "data": {
        "@type": "types.googleapis.com/hashicorp.consul.foo.v1alpha1.Bar",
        "baz": "Hello World"
      }
    }
  }
EOF
```

## Validation

Broadly, there are two kinds of validation you might want to perform against
your resources:

- **Structural** validation ensures the user's input is well-formed, for
  example: checking that a required field is provided, or that a port is within
  an acceptable range.
- **Semantic** validation ensures that the resource makes sense in the context
  of *other* resources, for example: checking that an L7 intention is not
  targeting an L4 service.

Structural validation should be done up-front, before the resource is admitted,
using a validation hook provided in the type registration:

```Go
func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:     pbv1alpha1.BarType,
		Proto:    &pbv1alpha1.Bar{}, 
		Scope:    resource.ScopeNamespace,
		Validate: validateBar,
	})
}

func validateBar(res *pbresource.Resource) error {
	var bar pbv1alpha1.Bar
	if err := res.Data.UnmarshalTo(&bar); err != nil {
		return resource.NewErrDataParse(&bar, err)
	}
	if bar.Baz == "" {
		return resource.ErrInvalidField{
			Name:    "baz",
			Wrapped: resource.ErrMissing,
		}
	}
	return nil
}
```

Semantic validation should be done asynchronously, after the resource is
written, by controllers ([covered below](#controllers)).

## Authorization

You can control how operations on your resource type are authorized by providing
a set of ACL hooks:

```Go
func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:  pbv1alpha1.BarType,
		Proto: &pbv1alpha1.Bar{}, 
		Scope: resource.ScopeNamespace,
		ACLs: &resource.ACLHooks{,
			Read:  authzReadBar,
			Write: authzWriteBar,
			List:  authzListBar,
		},
	})
}

func authzReadBar(authz acl.Authorizer, authzContext *acl.AuthorizerContext, id *pbresource.ID,  _ *pbresource.Resource) error {
	return authz.ToAllowAuthorizer().
		BarReadAllowed(id.Name, authzContext)
}

func authzWriteBar(authz acl.Authorizer, authzContext *acl.AuthorizerContext, res *pbresource.Resource) error {
	return authz.ToAllowAuthorizer().
		BarWriteAllowed(res.ID().Name, authzContext)
}

func authzListBar(authz acl.Authorizer, authzContext *acl.AuthorizerContext) error {
	return authz.ToAllowAuthorizer().
		BarListAllowed(authzContext)
}
```

If you do not provide ACL hooks, `operator:read` and `operator:write`
permissions will be required.

## Mutation

Sometimes, it's necessary to modify resources before they're persisted. For
example, to set sensible default values or normalize user input. You can do this
by providing a mutation hook:

```Go
func RegisterTypes(r resource.Registry) {
	r.Register(resource.Registration{
		Type:   pbv1alpha1.BarType,
		Proto:  &pbv1alpha1.Bar{}, 
		Scope:  resource.ScopeNamespace,
		Mutate: mutateBar,
	})
}

func mutateBar(res *pbresource.Resource) error {
	var bar pbv1alpha1.Bar
	if err := res.Data.UnmarshalTo(&bar); err != nil {
		return resource.NewErrDataParse(&bar, err)
	}
	bar.Baz = strings.ToLower(bar.Baz)
	return res.Data.MarshalFrom(&bar)
}
```

## Controllers

Controllers are where the business logic of your resources will live. They're
asynchronous [reconciliation loops] that "wake up" whenever a resource is
modified to validate and realize the changes.

You can create a new controller using the [builder API]. Start by identifying
the resource type you want this controller to manage, and provide a reconciler
that will be called whenever a resource of that type is changed.

```Go
package foo

import (
	"context"

	"github.com/hashicorp/consul/internal/controller"
	pbv1alpha1 "github.com/hashicorp/consul/proto-public/pbfoo/v1alpha1"
	"github.com/hashicorp/consul/proto-public/pbresource"
)

func barController() controller.Controller {
	return controller.NewController("bar", pbv1alpha1.BarType).
		WithReconciler(barReconciler{})
}

type barReconciler struct{}

func (barReconciler) Reconcile(ctx context.Context, rt controller.Runtime, req controller.Request) error {
	rsp, err := rt.Client.Read(ctx, &pbresource.ReadRequest{Id: req.ID})
	switch {
	case status.Code(err) == codes.NotFound:
		return nil
	case err != nil:
		return err
	}

	var bar pbv1alpha1.Bar
	if err := rsp.Resource.Data.UnmarshalTo(&bar); err != nil {
		return err
	}
	rt.Logger.Debug("Hello from bar reconciler!", "baz", bar.Baz)

	return nil
}
```

[reconciliation loops]: https://www.oreilly.com/library/view/97-things-every/9781492050896/ch73.html
[builder API]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#Controller

Next, register your controller with the controller manager. Another common
pattern is to have your package expose a method for registering controllers,
which is called from `registerControllers` in [`server.go`].

[`server.go`]: ../../../agent/consul/server.go

```Go
package foo

func RegisterControllers(mgr *controller.Manager) {
	mgr.Register(barController())
}
```

```Go
package consul

func (s *Server) registerControllers() {
	// …
	foo.RegisterControllers(s.controllerManager)
	// …
}
```

### Retries

By default, if your reconciler returns an error, it will be retried with
exponential backoff. While this is correct in most circumstances, you can
override it by returning [`RequeueAfter`] or [`RequeueNow`].

[`RequeueAfter`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#RequeueAfter
[`RequeueNow`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#RequeueNow

```Go
func (barReconciler) Reconcile(context.Context, controller.Runtime, controller.Request) error {
	if time.Now().Hour() < 9 {
		return controller.RequeueAfter(1 * time.Hour)
	}
	return nil
}
```

### Status

Controllers can communicate the result of reconciling resource changes (e.g.
surfacing semantic validation issues) with users and other controllers by
updating the resource's status using the `WriteStatus` method.

Each resource can have multiple statuses, typically one per controller,
identified by a string key. Statuses are composed of a set of conditions, which
represent discreet observations about the resource in relation to the current
state of the system.

That all sounds a little abstract, so let's take a look at an example.

```Go
client.WriteStatus(ctx, &pbresource.WriteStatusRequest{
	Id:     res.Id,
	Key:    "consul.io/bar",
	Status: &pbresource.Status{
		ObservedGeneration: res.Generation,
		Conditions: []*pbresource.Condition{
			{
				Type:    "Healthy",
				State:   pbresource.Condition_STATE_TRUE,
				Reason:  "OK",
				Message: "All checks are passing",
			},
			{
				Type:    "ResolvedRefs",
				State:   pbresource.Condition_STATE_FALSE,
				Reason:  "INVALID_REFERENCE",
				Message: "Bar contained an invalid reference to qux",
				Resource: resource.Reference(bar.Qux, ""),
			},
		},
	},
})
```

In the previous example, the controller makes two observations about the
current state of the resource:

1. That it's "healthy" (whatever that means in this hypothetical scenario)
1. That it contains a reference that couldn't be resolved

The `Type` and `Reason` should be simple, machine-readable, strings, but there
aren't any strict rules about what are acceptable values. Over time, we
anticipate that common values will emerge that we'll standardize on for
consistency.

`Message` should be a human-readable explanation of the condition.

> **Warning**  
> Writing a status to the resource will cause it to be re-reconciled. To avoid
> infinite loops, we recommend dirty checking the status before writing it with
> [`resource.EqualStatus`].

[`resource.EqualStatus`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/resource#EqualStatus

### Watching Other Resources

In addition to watching their "managed" resources, controllers can also watch
resources of different, related, types. For example, the service endpoints
controller also watches workloads and services.

```Go
func barController() controller.Controller {
	return controller.NewController("bar", pbv1alpha1.BarType).
		WithWatch(pbv1alpha1.BazType, controller.MapOwner)
		WithReconciler(barReconciler{})
}
```

The second argument to `WithWatch` is a [dependency mapper] function. Whenever a
resource of the watched type is modified, the dependency mapper will be called
to determine which of the controller's managed resources need to be reconciled.

[`dependency.MapOwner`] is a convenience function which causes the watched
resource's [owner](#ownership--cascading-deletion) to be reconciled.

[dependency mapper]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#DependencyMapper
[`dependency.MapOwner`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller/dependency#MapOwner

### Placement

By default, only a single, leader-elected, replica of each controller will run
within a cluster. Sometimes it's necessary to override this, for example when
you want to run a copy of the controller on each server (e.g. to apply some
configuration to the server whenever it changes). You can do this by changing
the controller's placement.

```Go
func barController() controller.Controller {
	return controller.NewController("bar", pbv1alpha1.BarType).
		WithPlacement(controller.PlacementEachServer)
		WithReconciler(barReconciler{})
}
```

> **Warning**  
> Controllers placed with [`controller.PlacementEachServer`] generally shouldn't
> modify resources (as it could lead to race conditions).

[`controller.PlacementEachServer`]: https://pkg.go.dev/github.com/hashicorp/consul/internal/controller#PlacementEachServer

## Ownership & Cascading Deletion

The resource service implements a lightweight `1:N` ownership model where, on
creation, you can mark a resource as being "owned" by another resource. When the
owner is deleted, the owned resource will be deleted too.

```Go
client.Write(ctx, &pbresource.WriteRequest{
	Resource: &pbresource.Resource{,
		Owner: ownerID,
		// …
	},
})
```