Including external module dependencies in existing timoni module

[rrmistry]

Business problem being solved for

🙏 My apologies in advance if I did not understand any timoni concepts correctly.

Currently, we have a large helm chart that includes external dependencies as sub-charts (e.g. a helm chart for databases, another for controllers, another for operators, etc.)

We are attempting to convert this helm chart into timoni module / bundle analogues.

The problem is: How to reliably include external dependencies as part of a timoni module?

Currently we use Helm sub-charts to include external dependencies as part of our helm chart.

Example my-apps Helm chart for reference

# my-apps/Charts.yaml
apiVersion: v2
name: my-apps
description: Helm Chart to deploy my app resources
type: application
version: 0.1.0
appVersion: 1.0.0

# Third party (external) dependencies
dependencies:

  - condition: redis.enabled
    name: redis
    repository: oci://registry-1.docker.io/bitnamicharts
    version: "20.0.3"

  - condition: postgresql.enabled
    name: postgresql
    repository: oci://registry-1.docker.io/bitnamicharts
    version: "15.5.27"

---
# my-apps/templates/*.yaml

# ... classic helm chart template files for our app resources ...
...
# my-apps/templates/deployment.yaml
# my-apps/templates/ingress.yaml
# my-apps/templates/service.yaml
# my-apps/templates/secret.yaml
...

---
# my-apps/values.yaml

# ... my helm chart values ...
valueA: 123
valueB: def

# Third party (external) dependency's values
redis:
  enabled: true
  architecture: "standalone"
  auth:
    enabled: true
    username: default
    password: ********

postgresql:
  enabled: true
  architecture: "standalone"
  auth:
    enablePostgresUser: true
    postgresPassword: ********
    database: myapps
    username: myapps-user
    password: ********

The beauty of the helm chart approach is that we can reliably build or deploy our app and their dependencies as part of the same helm chart with single commands.

# Build
helm template --dependency-update --dry-run=server my-apps ./my-apps/

# Deploy
helm install my-apps ./my-apps/ -f ./my-apps/values.yaml -f overrides.yaml ...

Known workarounds

With Timoni, as I understand it so far, the same end-result requires multiple steps:

• Build:

  1. Create your app's resources as part of a timoni module

     📝 Write CUE files ...

  2. Push your timoni module as an artifact to a container registry

     timoni mod push ...

  3. Create a timoni bundle that includes your module as part of this bundle

     📝 Write CUE files ...

  4. Include external dependencies as part of the same timoni bundle (e.g. redis, postgres, etc.)

     📝 Write CUE files ...

  5. Push this timoni bundle to a container registry

     timoni artifact push ...

• Deploy:

  1. Create a new timoni bundle that includes your combined timoni bundle from the build step (as a single timoni instance in this bundle)

     📝 Write CUE files ...

  2. Deploy timoni bundle to a Kubernetes cluster

     timoni bundle apply ...

     ✅ This will deploy your apps and their dependencies reliably as a single deployment

This observation was based on the end-to-end workflow described in the timoni e2e.yaml workflow file.

Problem with workaround

This approach requires multiple steps to get to a clean singular timoni bundle that can achieve the same end-result (deploying our app and its dependencies reliably and together).

Best solution (so far)

The cleanest solution I could find so far was to use Flux's CRDs and deploy multiple custom resources for HelmRepository and HelmRelease type that would include the external dependencies as part our module.

But this approach itself has a dependency on having Flux installed on the cluster beforehand.

Conclusion

It is still ideal to have timoni allow including an external bundle within a module as an external dependency (not requiring the cluster to have Flux already installed).

But I'm not sure if this is a valid feature request or if this is possible but I don't know how to do it yet.

[rrmistry]

I think I just found the ideal solution. I don't know why I didn't see it in the docs earlier.

timoni bundle syntax supports local folder as a module reference.

Referencing a local module folder means I don't need 2 bundle files, just 1, and I don't need to push and pull OCI artifacts.

Below is a what I got working with local folder reference:

// my-apps/bundles/my-apps-for-a-clean-cluster.cue
bundle: {
    apiVersion: "v1alpha1"
    name:       "my-apps"

    // timoni will deploy the following instances in the order they are defined below
    instances: {

        // flux is necessary to deploy first because it will deploy HelmRepository and HelmRelease CRDs
        // these CRDs are used later to allow Flux to deploy our external dependencies
        "flux": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-aio"
            namespace: "flux-system"
            values: {
                controllers: {
                    helm: enabled:         true
                    kustomize: enabled:    false
                    notification: enabled: false
                }
                hostNetwork:     false
                securityProfile: "privileged"
            }
        }

        // Install external dependencies
        "cert-manager": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
            namespace: "cert-manager"
            values: {
                repository: url: "https://charts.jetstack.io"
                chart: {
                    name:    "cert-manager"
                    version: "1.x"
                }
                helmValues: {
                    installCRDs: true
                }
            }
        }
        // Install external dependencies
        "ingress-nginx": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
            namespace: "ingress-nginx"
            values: {
                repository: url: "https://kubernetes.github.io/ingress-nginx"
                chart: {
                    name:    "ingress-nginx"
                    version: "4.x"
                }
                helmValues: {
                    controller: service: type: "NodePort"
                }
            }
        }

        // Install my apps module (analogue of helm chart)
        "my-apps": {
            module: url: "file://../." // <<<<< Local folder reference (relative to current working directory)
            namespace: "my-apps"
            values: {}
        }
    }
}

However, I did notice that with this approach, the external dependencies of cert-manager and ingress-nginx used above do not get renamed as part of my parent release (like how it does for helm sub-charts).

This is not a deal breaker for us, because we can still add flux to our bundle like above, but then create the custom resources for HelmRepository and HelmRelease in our module but rename them as needed (e.g. "ingress-nginx" would become #config.metadata.name + "ingress-nginx")

[rrmistry]

@salotz RE:

Why wrap the chart in your own module?

I'm struggling to get feature parity with Helm sub-charts. Specifically, the feature that when including a sub-chart in Helm, then those sub-charts get the Helm Release name prefix.

E.g. Sub-chart statefulsets get the name <helm-release-name>-redis-master or <helm-release-name>-postgresql, etc.

This is preferred because it keeps the namespace cleaner and more organized when working in a shared namespace with shared bundles.

At the bundle level, I can't figure out how to parameterize this prefix such that different teams can re-use our bundle but with their own prefixes (e.g. dependency resources get the name a-team-redis-master, b-team-postgresql, etc.)

The best I could find was to have each team manage their own bundle files where these helm release names are hard-coded:

// ./bundles/a-team.cue
bundle: {
    apiVersion: "v1alpha1"
    name:       "cluster-addons"
    instances: {
        "a-team-redis": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
            namespace: "shared-system"
            values: {
                repository: url: "https://charts.jetstack.io"
                chart: {
                    name:    "redis"
                    version: "20.x"
                }
                helmValues: {
                    // ... a-team specific values (most are the same as b-team)
                }
            }
        }
        "a-team-postgresql": {
            // ... a-team specific config and values but for postgresql
        }
        "a-team-apps": {
            module: url: "oci://registry-1.docker.io/my-company/modules/my-timoni-bundle"
            namespace: "shared-system"
            values: {
                repository: url: "https://charts.jetstack.io"
                chart: {
                    name:    "my-timoni"
                    version: "1.x"
                }
                helmValues: {
                    // ... a-team specific values (most are the same as b-team)
                }
            }
        }
    }
}

// ----------------------------------------------------------------------------
// ./bundles/b-team.cue
bundle: {
    apiVersion: "v1alpha1"
    name:       "cluster-addons"
    instances: {
        "b-team-redis": {
            module: url: "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
            namespace: "shared-system"
            values: {
                repository: url: "https://charts.jetstack.io"
                chart: {
                    name:    "redis"
                    version: "20.x"
                }
                helmValues: {
                    // ... b-team specific values (most are the same as a-team)
                }
            }
        }
        "b-team-postgresql": {
            // ... b-team specific config and values but for postgresql
        }
        "b-team-apps": {
            module: url: "oci://registry-1.docker.io/my-company/modules/my-timoni-bundle"
            namespace: "shared-system"
            values: {
                repository: url: "https://charts.jetstack.io"
                chart: {
                    name:    "my-timoni"
                    version: "1.x"
                }
                helmValues: {
                    // ... b-team specific values (most are the same as a-team)
                }
            }
        }
    }
}

The above feels like a lot of redundant code at the bundle level between teams.

I'm open to suggestions on how to parameterize this prefix, or what the best way to approach this is.

[salotz]

You can parametrize the names with cue and have those names passed in by each team when they apply the bundle.

[rrmistry]

Thanks @salotz!

After some learning of CUE, I got this setup to work:

> timoni bundle build --file "bundles/podinfo.cue" --file "bundles/team_a.cue"

Where file contents were:

// ./bundles/podinfo.cue
bundle: {
    apiVersion: "v1alpha1"
    name:       string | *"podinfo"

    _namespace: string | *"default"

    instances: {
        "\(name)-redis": {
            module: {
                url:     "oci://ghcr.io/stefanprodan/modules/redis"
                version: "7.2.4"
            }
            namespace: _namespace
            values: maxmemory: int | *256
        }
        "\(name)": {
            module: url:     "oci://ghcr.io/stefanprodan/modules/podinfo"
            module: version: "6.5.4"
            namespace: _namespace
            values: caching: {
                enabled:  true
                redisURL: "tcp://redis:6379"
            }
        }
    }
}

// ----------------------------------------------------------------------------

// ./bundles/team_a.cue
bundle: {
    name: "team-a"

    _namespace: "shared-system"

    instances: {
        "\(name)-redis": {
            values: maxmemory: 123
        }
    }
}

But something I still can't figure out is how to tap into the command line parameter for the namespace from the timoni bundle build command.

Running below:

> timoni bundle build --file "bundles/podinfo.cue" --file "bundles/team_a.cue" --namespace "team-b"

Still deploys the resources in the namespace defined in bundles/team_a.cue. I can't figure out how to replace the value for _namespace variable with the timoni command line parameter.

[rrmistry]

After some more deliberations and looking at GetBundle in internal/engine/bundle_builder.go and how it is used in cmd/timoni/bundle_build.go, it feels that the CUE configuration is preferred at the bundle level, and the --namespace flag has no effect.

So the cleanest solution now is to utilize some CUE and timoni techniques to allow the namespace to be configurable at the bundle level.

With below file:

// ./bundles/podinfo.cue
bundle: {
    ...
    // Same as before but with a small change
    _namespace: string | *"default" @timoni(runtime:string:NAMESPACE)
    ...
}

// ----------------------------------------------------------------------------

// ./bundles/team_a.cue
bundle: {
    ...
    // // Comment out or remove hard-coded namespace from this file
    // _namespace: "shared-system"
    ...
}

I can now use either options:

# Pass namespace via stdin
echo 'bundle: {_namespace: "team-b"}' | timoni bundle build --file podinfo.cue --file team_a.cue -f -

# Pass namespace via environment variable
export NAMESPACE="team-b";
timoni bundle build --file podinfo.cue --file team_a.cue --runtime-from-env

This is enough for me to be satisfied for my use case.

Thanks for the engagement @salotz! This was a nice learning experience.

[salotz]

You should also be able to use the bundle values file to pass in config as well. That vs runtime environment is just up to how you do config.

[rrmistry]

Sharing syntax here for anyone who may need to include Flux CRDs in their module:

package templates

import (
    timoniv1 "timoni.sh/core/v1alpha1"
    flux_helmrelease "helm.toolkit.fluxcd.io/helmrelease/v2"
)

// Reference: https://fluxcd.io/flux/components/helm/helmreleases/
#BitnamiRedisHelmRelease: flux_helmrelease.#HelmRelease & {
    #config: #Config

    // **********************************************
    // I was looking for elegant syntax like this:
    metadata: timoniv1.#MetaComponent & {
        #Meta:      #config.metadata
        #Component: "redis"
    }
    // **********************************************

    spec: flux_helmrelease.#HelmReleaseSpec & {
        #config.bitnamiHelmRepository.redis.releaseSpec
    }
}