Multitenancy setup using helm

[arsanysamuel]

I'm trying to have a multitenant setup using dns based multitenancy
Currently I'm creating tenants manually by creating a new site, manually migrating and adding a dns entry.
It mentions using the bench setup nginx command to generate the an nginx config file, the generated config file includes server_name which has all the site names/domains.
To adapt with the frappe helm charts and frappe's docker image, as it's using nginx_entrypoint.sh to generate a config file based on nginx-template.conf, I've modified line 11 to be server_name .${FRAPPE_SITE_NAME_HEADER}; to have a dot before the site name header variable since I'm using subdomains for tenants.
The problem is that with or without this edit I always can access the first site created by the deployment and when I try to access any new site it returns an empty 200 response without any content, on a browser it downloads an empty file.
Am I missing something? Is there any guide for a multitenant setup using frappe helm?

[revant]

I've one helm release as a bench. It creates a service to be exposed by ingress or load balancer

• for each site create an ingress pointing to the service
• for custom domain use annotation to override host https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-nginx-upstream-vhost
• case where you've no ingress controller, directly expose the service as lb service and configure dns pointing to lb ip
• to have wildcard subdomain follow https://cert-manager.io/docs/configuration/acme/dns01 or place your wildcard cert in a secret and attach it to ingress.

If wildcard cert and wildcard dns is configured, you just need to use kubernetes api to add / remove site.

Check this https://discuss.frappe.io/t/manage-official-or-custom-frappe-benches-on-kubernetes/107793

[arsanysamuel]

Thank you for your answer, most probably I have the same setup that you have unless I don't use k8s_bench which might be a good option, currently I'm using the charts in this repo with few customizations.

I have properly configured domain dns and cert to have the hosts for my sites.
I tried creating an ingress pointing to the erpnext service, and I've edited the site's ingress to have this line in metadata.annotations:

    nginx.ingress.kubernetes.io/upstream-vhost: <site-host>

The site host here is a subdomain of the default site host, for instance if the default site host is dev.example.com the new site host is sub.dev.example.com

This resulted in having both the default and the new site domains point to the same site (default site created by the chart).

I have even tried having one ingress that has multiple hosts (dev.example.com, *.dev.example.com) and I got the same result.

Do you have any idea what could be the issue? Thanks in advance

[arsanysamuel]

SOLVED

Using the charts in this repo with a frappe_docker custom image, for having multitenants with a domain example.com and subdomains *.example.com:

1. Add your domain and wildcard subdomain to your DNS and SSL certificate
2. Build a custom frappe docker image with a modified nginx-template.conf:
   1. Modify L11 to support subdomains:

       server_name .${FRAPPE_SITE_NAME_HEADER};
      

   2. Change every other ${FRAPPE_SITE_NAME_HEADER} to $host (Lines: 34, 41, 55, 58, 65)
3. Add an additional host for the wildcard subdomains in values.yaml, no need to create an additional ingress:

   ingress:
     ingressName: "example"
     hosts:
     - host: "example.com"
       paths:
       - path: /
         pathType: ImplementationSpecific
     - host: "*.example.com"
       paths:
       - path: /
         pathType: ImplementationSpecific
   

4. Enable DNS multitenancy: bench config dns_multitenant on
5. Create a new site using a subdomain as the name, include the --mariadb-user-host-login-scope "%" option so the new tenant works after new deployments and host IP changes:

   bench new-site <site-hostname> --mariadb-user-host-login-scope "%" --install-app erpnext