feat: Use CAPI for provisioning clusters (#424)

* Bump cluster-api-migration * bump migrator * Bump cluster-api-migration * bump migrator * bump migrator * bump migrator * Update GCP migration config * optimize imports * Remove --cluster-api flag * update google bootstrap flags * Fix deploy logic * bump migrator * update destroy bootstrap flags for google provider * Check if cluster exists * update destroy steps * Fix deploy * Add logging * Add missing new line * Fix log types * Add client ID and secret to init survey * remove cluster resources during destroy * Fix wait command * Remove plural clusters watch command * Run go mod tidy * Fix unit tests * Print step numbers for bootstrap and migration * Remove plural cluster watch command and some unused code * Remove build step and update descriptions for CAPI deploy * Refactor deploy and migration steps * Refactor destroy steps * Add destroy logs * Refactor * Move CAPI related logic from cmd to pkg * Extract common code * Move checks * Fix minor import issue * Cleanup * Remove unused flag Remove duplicated command * Minor improvements * Add TODO * add post install step * Update cluster readiness check * Fix merge conflicts * Update migration configuration for gcp * Export execute steps function * Refactor * Refactor migration * Add tests for common functions * Improve GCP preflight checks * Add tests for migration functions * Update messaging * add kind provider * Raise destroy timeout * Refactor cilium.go * Fix resource group and storage account name validation * Add command to check if chart is installed * save kubeconfig * add kind configuration * fix kind configuration * fix docker destroy * normilize kind * update e2e test * update github action * bump kind action * create bootstrap namespace * create bootstrap namespace * add extra debug * do not run migrate when cluster already migrated * read sa email from credentials file * add vendor dir to gitignore * fix import cycle * add PLURAL_DISABLE_MP_TABLE_VIEW env for machine pools view * remove bootstrap operator dependencies * cilium update * refactor * split e2e tests * change name * Refactor e2e workflows * distinguish between regular and cluster api * distinguish between regular and cluster api - fix * distinguish between regular and cluster api - improvement * distinguish between regular and cluster api - improvement * distinguish between regular and cluster api - improvement * add e2e test for cluster api * enable list view for destroy * add e2e test to check installed packages * fix linter * Update github.com/gin-gonic/gin to avoid CVE * Bump dependencies * Read Go version from go.mod in CI * Bump dependencies * improve error handling for deoploy/destroy cluster * e2e update machine pool * Refactor storage account code * Fixes * Fix unit tests * Fix kind delete * remove role permissions check for gcp SA and use local CLI ADC for migration and bootstraping * fetch AvailabilityZones * fix unit test * Use Microsoft Graph SDK to create service principal and get client ID and secret * set bootstrapMode flag for gcp during the bootstrap phase * fix fetching zones * Add proper role assignment to Azure service principal * fix execute not showing error and add workaround for tf value templating issue * Minor improvements * read gcp credentials from adc file * Fix client ID * change migrate to run deploy at the end and run gcp in bootstrapMode during migrate * update gcp permissions check * set azure bootstrap mode flag Signed-off-by: David van der Spek <[email protected]> * Add commit flag at the end of running migrate (#436) It's very likely a large number of users will forget to manage their git, we should just remove that possibility w/ this. * do not use bootstrap mode for the gcp migration * improve gcp permissions check messaging * Fix typo * Enable OIDC issuer for Azure clusters * add some todo comments Signed-off-by: David van der Spek <[email protected]> * Create temporary service principal with password during deploy and destroy * Refactor * e2e update machine pool version * Fix destroy * update bootstrap step building logic * add plural build-values REPO * init bubbletea tui * revert bootstrap step changes * Resolve Helm issue * Extract methods from bootstrap steps * Fix destroy * Modify aws auth configmap manually to solve migration chicken-egg (#437) * Modify aws auth configmap manually to solve migration chicken-egg This allows us to reusably modify the aws-auth configmap for eks from the client which should help resolve some migrration-time issues * add to migrate steps * Add secret list and create funcs * Add kube initializer with context * add feature flag for CAPI stuff * fix build * Add kube initializer with context * set aws credentials * cleanup build values command * use dynamic credentials for GCP without storing them on the repo * lint fix * Refactor * Rename file * allow overriding enable field of helm modules * Fix var name * Simplify migration * Restore uninstall azure-identity package step * update gcp permissions check name * fix nil pointer error when listing uninstalled package * improve fetching AZs * bump migrator version (#440) Signed-off-by: David van der Spek <[email protected]> * fix gcp provider name * remove credentials * Properly normalize Google -> GCP provider name and add migration step to update google provider name to gcp * update go.sum * make genmock * Fix executor println (#443) This was always saying "actionName <app>" instead of the passed action name. * bump migrator * small refactor * Bump migrator version * fix null replacment * Deprecate values.yaml migration * bump migrator * Fix Azure destroy after migration * Refactor step filtering * Fix Azure identity bug * add posthog feature call timeout and fix caching * cleanup some steps * Switch google to gcp during init * Update messaging for GCP * bump migrator * update go.sum * fix linters * ci: ensure docker buildx removes the running nodes (#448) Signed-off-by: David van der Spek <[email protected]> * Add semver validation for required bootstrap tf/helm modules on migration (#445) There are now some requirements for performing a migration tied to our helm/tf. This will at least guarantee they're installed at migrate time. * remove default values from migration values.yaml * go mod tidy * update AZs during migration * disable external-dns and plural-certmanager-webhook * Do not delete bootstrap cluster on failed deploy * fix disabling plural-certmanager-webhook Signed-off-by: David van der Spek <[email protected]> * also disable external dns on gcp and azure Signed-off-by: David van der Spek <[email protected]> * Update step handling * Add retry mechanism * Fix step numbering * Fix unit tests * Further improvements * Use map to store provider tags * add move state backup and restore to capi deploy * Further improvements * Fix OIDC issuer step * Fix typo * add initial step confirm support * move capi backup to .plural dir and add multi-cluster backup support * Remove tui package * add conditional recovery steps when cluster issues are detected --------- Signed-off-by: David van der Spek <[email protected]> Co-authored-by: Lukasz Zajaczkowski <[email protected]> Co-authored-by: Sebastian Florek <[email protected]> Co-authored-by: David van der Spek <[email protected]> Co-authored-by: michaeljguarino <[email protected]> Co-authored-by: David van der Spek <[email protected]>
pluralsh · Sep 26, 2023 · afbd9e6 · afbd9e6
1 parent 4bd34b5
commit afbd9e6
Show file tree

Hide file tree

Showing 96 changed files with 6,564 additions and 872 deletions.
diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
@@ -28,7 +28,7 @@ jobs:
           fetch-depth: 0
       - uses: actions/setup-go@v3
         with:
-          go-version: 1.18
+          go-version-file: go.mod
       - uses: github/codeql-action/init@v2
         with:
           languages: ${{ matrix.language }}

diff --git a/.github/workflows/e2e.yaml b/.github/workflows/e2e.yaml
@@ -4,13 +4,53 @@ on:
     branches:
       - main
 jobs:
+  create-cluster-capi:
+    name: Create cluster with Cluster API
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Create kind cluster
+        uses: helm/[email protected]
+        with:
+          install_only: true
+      - run: |
+          wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
+          echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
+          sudo apt-get update
+          sudo apt-get install -y terraform
+      - run: |
+          GOBIN="$HOME"/bin make build-cli
+          chmod 755 plural.o
+          mv plural.o /usr/local/bin/plural
+      - run: hack/e2e/kind-install-for-capd.sh
+      - run: hack/e2e/setup-plural.sh
+        env:
+          CLI_E2E_CONF: ${{ secrets.CLI_E2E_CONF }}
+          CLI_E2E_IDENTITY_FILE: ${{ secrets.CLI_E2E_IDENTITY_FILE }}
+          CLI_E2E_KEY_FILE: ${{ secrets.CLI_E2E_KEY_FILE }}
+          CLI_E2E_PUBLIC_KEY: ${{ secrets.CLI_E2E_PUBLIC_KEY }}
+          CLI_E2E_PRIVATE_KEY: ${{ secrets.CLI_E2E_PRIVATE_KEY }}
+          CLI_E2E_SHARING_PRIVATE_KEY: ${{ secrets.CLI_E2E_SHARING_PRIVATE_KEY }}
+          CLI_E2E_SHARING_PUBLIC_KEY: ${{ secrets.CLI_E2E_SHARING_PUBLIC_KEY }}
+          USE_CLUSTER_API: true
+      - run: go test -v -race ./pkg/test/e2eclusterapi/... -tags="e2e"
+      - run: |
+          cd $HOME/test
+          plural destroy --force --all --commit=""
+        env:
+          PLURAL_DESTROY_CONFIRM: true
+          PLURAL_DESTROY_AFFIRM_UNINSTALL_APPS: true
+          PLURAL_DISABLE_MP_TABLE_VIEW: true
   create-cluster:
+    if: false
+    name: Create cluster
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
         uses: actions/checkout@v3
-      - name: Create k8s Kind Cluster
-        uses: helm/kind-action@v1.5.0
+      - name: Create kind cluster
+        uses: helm/kind-action@v1.8.0
         with:
           install_only: true
       - run: |
@@ -31,10 +71,12 @@ jobs:
           CLI_E2E_PRIVATE_KEY: ${{ secrets.CLI_E2E_PRIVATE_KEY }}
           CLI_E2E_SHARING_PRIVATE_KEY: ${{ secrets.CLI_E2E_SHARING_PRIVATE_KEY }}
           CLI_E2E_SHARING_PUBLIC_KEY: ${{ secrets.CLI_E2E_SHARING_PUBLIC_KEY }}
+          INSTALL_APP: console
+          INSTALL_RECIPE: console-kind
       - run: go test -v -race ./pkg/test/e2e/... -tags="e2e"
       - run: |
           cd $HOME/test
           plural destroy --force --all --commit=""
         env:
           PLURAL_DESTROY_CONFIRM: true
-          PLURAL_DESTROY_AFFIRM_UNINSTALL_APPS: true
+          PLURAL_DESTROY_AFFIRM_UNINSTALL_APPS: true
diff --git a/.gitignore b/.gitignore
@@ -1,10 +1,21 @@
-# Binaries for programs and plugins
+# IDE files
+.idea/
+.vscode/
+
+# Testing files
+context.yaml
+workspace.yaml
+
+# Build files and binaries
+dist/
+build/
 *.exe
 *.exe~
 *.dll
 *.so
 *.o
 *.dylib
+__debug_bin
 
 # Test binary, built with `go test -c`
 *.test
@@ -13,15 +24,6 @@
 *.out
 forge*.o
 plural*.o
-# Dependency directories (remove the comment below to include it)
-# vendor/
 
-# IDE dirs
-.idea/
-
-dist/
-build/
-
-# Testing files
-context.yaml
-workspace.yaml
+# Vendored dependencies
+vendor/
diff --git a/Makefile b/Makefile
@@ -37,8 +37,8 @@ git-push:
 	git push
 
 .PHONY: install
-install: build-cli-ui
-	mv $(OUTFILE) ~/bin/plural
+install:
+	go install -ldflags '$(LDFLAGS)' .
 
 .PHONY: build-cli
 build-cli: ## Build a CLI binary for the host architecture without embedded UI

diff --git a/cmd/plural/api.go b/cmd/plural/api.go
@@ -1,10 +1,11 @@
 package plural
 
 import (
-	"github.com/pluralsh/plural/pkg/api"
-	"github.com/pluralsh/plural/pkg/utils"
 	"github.com/pluralsh/polly/algorithms"
 	"github.com/urfave/cli"
+
+	"github.com/pluralsh/plural/pkg/api"
+	"github.com/pluralsh/plural/pkg/utils"
 )
 
 func (p *Plural) apiCommands() []cli.Command {
@@ -111,7 +112,7 @@ func (p *Plural) handleCharts(c *cli.Context) error {
 
 func (p *Plural) handleTerraforma(c *cli.Context) error {
 	p.InitPluralClient()
-	tfs, err := p.GetTerraforma(c.Args().First())
+	tfs, err := p.GetTerraform(c.Args().First())
 	if err != nil {
 		return api.GetErrorResponse(err, "GetTerraforma")
 	}