Argo setup for multiples teams with separate env

[s1apped]

Hi,

We are thinking to use Argo for managing our test environments. The idea is to have for each team having it's own environment in separate namespace. I'm wondering what would be the best approach to handle this scenario? Would it be something like having single repository with branch for each team containing configured applications or sub directories? Worth to mention is that all services are having versioned helm charts in separate repository?

Looking forward to your feedback! Thanks.

[cnukwas]

I suppose, you could use a single mono repo with folders for each application or service, then use that path in the Argo Application set up. Ideally it's better if each team has their own repo to avoid issues, but this should work too.
You may also want to look at Argo ApplicationSet controller since that will help to automate the Application creation, based on the available generators.

[meier-christoph]

Hi,

this is how we do this:

• we have 1 cluster per env
• each cluster has its own argocd instance
• we deploy in-cluster only, envs can‘t connect (diff networks)
• we have one cluster with argocd that manages the other clusters i.e we use argocd to install the other argocds
• each team has one or many namespaces per cluster
• each team has one github repo, we had one mono repo first but its hard to do acces control, multiple repos is the way
• in that repo we have a branch for each env i.e cluster
• first folder is namespace, then app
• we don‘t use helm but something similar, in our case we have some resources in yaml and a separate file for just the images, the file with the images is generated by our ci/cd and commited using the github api, the yaml files are the config for the app, secrets are stored using sealed secrets or generated by an operator in the cluster

we used applicationset first but then created our own plugin, we use that one to generate the applications which then generate the actual resources. The pb we had with applicationsets is that we don‘t have control over labels of applications it creates and creating our own was easy.

you may also consider:

• whenever we build a new release we commit to staging and also open a pr to production (branches in repo), we only have to touch the images file, with helm you have it all in values which may be an issue
• the deploy to prod is done by the pr
• thus we have a lot of prs open (currently over 13k on the first mono repo, also a reason why you should not use a single repo)
• we use bots to cleanup and manage those pr (with probot) e.g closing pr if new minor version is created (using semver)

We are kind of happy with this setup, devs can add new apps without depending on an admin yet they have almost no access on the cluster itself, they do everything through argocd. Also, we can lose an entire cluster and re create it within a few hours, we did a DRP last month and it worked like a charm, most time is lost creating the vms and updating some networking rules (new cluster = new ips for lbs and ingress)

Cheers,
Chris

[cnukwas]

@meier-christoph thanks for sharing your set up and glad to hear that Argo is used extensively for admin and application tasks.
Not exactly related to this discussion, have you used any 3rd party or homegrown tooling for provisioning namespaces, applying RBAC to restrict access to team LDAP or AD groups or oauth integration?

Thanks

[meier-christoph]

We do not allow for namespaces to be created dynamically, if a team needs a new namespace it has to do it over support ticket and we configure the quota and rbac for that namespace which is then applied to the clusters using our infra argocd.

We are using carvel for some lighweight templating of yaml files to help with that.

For ldap we already had an existing tool, IGA from Evidian (which is now Atos), not sure I would recommend it though but it has been with us, well since forever and those things are hard to change. IGA manages the entire ldap not just the groups required for k8s. Basically, when someone is hired, HR adds them to IGA in a team and then there are rules which will add the user to all the groups required, including the groups which grant acces to the kube namespace of that team.

Anyway, here is their product site : https://www.evidian.com/products/identity-governance-and-administration-iga-10/

There is also some naming convention in use, again not related to k8s, it looks something like xxx_{cluster}_{namespace}_owner (the xxx is actually the IGA convention, its not important for kube though).

Now if we were to allow dynamic namespaces we would have to do things a bit differently. With a tool like https://kyverno.io/ you can generate resources conditionally e.g. on namespace creation you could then create roles and bindings but you would still have to figure out something that manages the ldap.

One possible solution would be to link namespaces with github orgs (we have an on prem github enterprise). If the user belongs to the orga he/she has access to the namespace which hosts these apps.