consul/internal
R.B. Boyer c029b20615
v2: ensure the controller caches are fully populated before first use (#20421)
The new controller caches are initialized before the DependencyMappers or the 
Reconciler run, but importantly they are not populated. The expectation is that 
when the WatchList call is made to the resource service it will send an initial 
snapshot of all resources matching a single type, and then perpetually send 
UPSERT/DELETE events afterward. This initial snapshot will cycle through the 
caching layer and will catch it up to reflect the stored data.

Critically the dependency mappers and reconcilers will race against the restoration 
of the caches on server startup or leader election. During this time it is possible a
 mapper or reconciler will use the cache to lookup a specific relationship and 
not find it. That very same reconciler may choose to then recompute some 
persisted resource and in effect rewind it to a prior computed state.

Change

- Since we are updating the behavior of the WatchList RPC, it was aligned to 
  match that of pbsubscribe and pbpeerstream using a protobuf oneof instead of the enum+fields option.

- The WatchList rpc now has 3 alternating response events: Upsert, Delete, 
  EndOfSnapshot. When set the initial batch of "snapshot" Upserts sent on a new 
  watch, those operations will be followed by an EndOfSnapshot event before beginning 
  the never-ending sequence of Upsert/Delete events.

- Within the Controller startup code we will launch N+1 goroutines to execute WatchList 
  queries for the watched types. The UPSERTs will be applied to the nascent cache
   only (no mappers will execute).

- Upon witnessing the END operation, those goroutines will terminate.

- When all cache priming routines complete, then the normal set of N+1 long lived 
watch routines will launch to officially witness all events in the system using the 
primed cached.
2024-02-02 15:11:05 -06:00
..
auth Change the multicluster group to v2 (#20430) 2024-02-01 12:08:26 -05:00
catalog Change the multicluster group to v2 (#20430) 2024-02-01 12:08:26 -05:00
controller v2: ensure the controller caches are fully populated before first use (#20421) 2024-02-02 15:11:05 -06:00
dnsutil feat(v2dns): add PTR query support (#20362) 2024-01-29 11:40:10 -05:00
go-sso [Cloud][CC-6925] Updates to pushing server state (#19682) 2023-12-04 10:25:18 -05:00
hcp hcp.v2.TelemetryState resource and controller implementation (#20257) 2024-01-31 14:47:05 -05:00
mesh Change the multicluster group to v2 (#20430) 2024-02-01 12:08:26 -05:00
multicluster Trigger the V1 Compat exported-services Controller when V1 Config Entries are Updated (#20456) 2024-02-02 15:30:04 -05:00
protohcl [Cloud][CC-6925] Updates to pushing server state (#19682) 2023-12-04 10:25:18 -05:00
protoutil mesh: compute more of the xRoute features into ComputedRoutes (#18980) 2023-09-22 16:13:24 -05:00
radix [COMPLIANCE] License changes (#18443) 2023-08-11 09:12:13 -04:00
resource Remove V2 PeerName field from pbresource.Tenancy (#19865) 2024-01-29 15:08:31 -05:00
resourcehcl mesh: rename Upstreams and UpstreamsConfiguration to Destinations* (#18995) 2023-09-25 12:03:45 -06:00
storage v2: ensure the controller caches are fully populated before first use (#20421) 2024-02-02 15:11:05 -06:00
tenancy Remove V2 PeerName field from pbresource.Tenancy (#19865) 2024-01-29 15:08:31 -05:00
testing Add Common Controller Caching Infrastructure (#19767) 2023-12-13 10:06:39 -05:00
tools NET-5824 Exported services api (#20015) 2024-01-23 10:06:59 +05:30