EVEREST-1811 | dynamically configure watches for controllers

[mayankshah1607]

CHANGE DESCRIPTION

Problem:
EVEREST-1811

Everest operator sometimes does not restart when new DBEngines and CRDs are installed. Due to this, the newly added APIs are not registered to scheme, the watches not configured, and the operator is indefinitely stuck in an error state.

Cause:

With the addition of Helm installation, we made it possible to install the core components and DBNamespaces separately. One issue this brought was that the Everest operator now needed to restart so that the newly added CRDs are registered to the client scheme, and the watchers can be configured during bootstrap. So we added some mechanism to detect if a new DBEngine is installed, and restart the pod.

But this mechanism fails sometimes if there are delays in the CRD creation, which can lead to a race condition. For example, if some CRDs are created late, the operator will restart regardless, and can fail to observe those CRDs. As a result, the watchers are only partially configured. Moreover, the restart logic checks only the GroupVersion in the scheme and assumes that all Kinds are registered.

Solution:

This PR removes the need to restart the Everest operator by dynamically registering watchers based on the installed DBEngines:

• We register the PXC, PSMDB and PG schemes to the controller during boot up, regardless of whether its DBEngine is installed or not. This is done to simplify code since it poses no harm.
• We use the underlying Controller interface exposed by controller-runtime to add watchers (to other controllers) during DBEngine reconciliation
• Each of the controllers (db, dbb, dbr) expose a ReconcileWatchers function that updates its watchers based on the DBEngines. This ReconcileWatchers is called during the DBEngine reconciliation

With such a mechanism, the various watchers are constantly being reconciled by the DBEngine controller. Such an approach requires no restart since the watchers are configured dynamically during runtime, and also makes no assumptions based on the scheme's registered groups (rather it ensures that the correct watchers are configured)

tl;dr - DBEngines reconciler reconciles the watchers on other reconcilers

CHECKLIST

Helm chart

• Is the helm chart updated with the new changes? (if applicable)

Jira

• Is the Jira ticket created and referenced properly?

Tests

• Is an Integration test/test case added for the new feature/change?
• Are unit tests added where appropriate?

[oksana-grishchenko]

will it work if initially any of the operators have not been installed?

[mayankshah1607]

Yes, the scheme only holds a mapping between the APIs/GVK and the Go structs to facilitate encoding/decoding responses from the Kube API server. It does not talk to Kubernetes

[recharte]

This is so much more elegant than killing the pod 🙈 congratulations 👏