release 0.49.0

CHANGELOG.rst

@@ -1,5 +1,47 @@
 .. _changelog:
 
+0.49.0
+------
+
+The release contains bug fixes to renku core service related to project migration.
+
+This release also contains initial support for next generation 'Renku 2.0' functionality. However,
+Renku 2.0 is still in early development and is not yet accessible to users. For more information,
+see our [roadmap](https://github.com/SwissDataScienceCenter/renku-design-docs/blob/main/roadmap.md).
+
+**Note for administrators**: this release includes breaking changes due to upgrading PostgreSQL to 16.1.0.
+This requires modifying the values file to work with the new PostgreSQL Helm chart.
+Please check (`the helm chart values changelog <https://github.com/SwissDataScienceCenter/renku/blob/master/helm-chart/values.yaml.changelog.md>`_)
+for detailed instructions.
+
+User-Facing Changes
+~~~~~~~~~~~~~~~~~~~
+
+**🐞 Bug Fixes**
+
+- **Core Service**: Fix issue with having to run project migration twice to migrate the Dockerfile/project template.
+
+Internal Changes
+~~~~~~~~~~~~~~~~
+
+**New Features**
+
+- **Data services**: Initial support for Renku 2.0 projects (alpha release)
+
+**Improvements**
+
+- **csi-rclone**: added rclone logs to regular node-plugin logs.
+  (`#11 <https://github.com/SwissDataScienceCenter/csi-rclone/pull/11>`_).
+
+
+Individual Components
+~~~~~~~~~~~~~~~~~~~~~
+
+- `renku-python 2.9.2 <https://github.com/SwissDataScienceCenter/renku-python/releases/tag/2.9.2>`_
+- `renku-data-services 0.5.0 <https://github.com/SwissDataScienceCenter/renku-data-services/releases/tag/v0.5.0>`_
+- `csi-rclone 0.1.7 <https://github.com/SwissDataScienceCenter/csi-rclone/releases/tag/v0.1.7>`_
+
+
 0.48.1
 ------
 
@@ -42,7 +84,8 @@ Internal Changes
 
 **Improvements**
 
-- **UI**: Add initial alpha implementation of Renku 1.0 projects
+- **Infrastructure**: Upgrade the version of PostgreSQL to 16.1.0.
+- **UI**: Add initial alpha implementation of Renku 2.0 projects
   (`#2875 <https://github.com/SwissDataScienceCenter/renku-ui/pull/2875>`_).
 
 Individual components
@@ -51,7 +94,6 @@ Individual components
 - `renku-ui 3.20.1 <https://github.com/SwissDataScienceCenter/renku-ui/releases/tag/3.20.1>`_
 - `renku-ui 3.20.0 <https://github.com/SwissDataScienceCenter/renku-ui/releases/tag/3.20.0>`_
 
-
 0.47.1
 ------
 
@@ -65,11 +107,11 @@ It doesn't bring any new features or bug fixes.
 This release expands Renku's cloud storage functionality in two key ways: First, mounted storages
 are now read **and write**, so you can use mounted storage as an active workspace for your data in a RenkuLab
 session. Second, we have expanded the cloud storage services you can integrate with RenkuLab. You can now
-mount not only S3 buckets, but also WebDAV-based storages and Azure Blobs. 
- 
+mount not only S3 buckets, but also WebDAV-based storages and Azure Blobs.
+
 If you use SSH sessions via the CLI, you can use cloud storage there too! Configure cloud storage for your
 project on RenkuLab.io, and those storages will be mounted in your remote session. Support for cloud
-storage in local Renku sessions is still on our roadmap. 
+storage in local Renku sessions is still on our roadmap.
 
 This release also adds the ability to change which resource class your session uses when you unpause the
 session, in case the original resource class is now full.
@@ -200,15 +242,15 @@ Internal Changes
 Individual components
 ~~~~~~~~~~~~~~~~~~~~~
 
-- `renku-python 2.8.2 <https://github.com/SwissDataScienceCenter/renku-python/releases/tag/2.8.2>`_ 
+- `renku-python 2.8.2 <https://github.com/SwissDataScienceCenter/renku-python/releases/tag/2.8.2>`_
 - `renku-ui 3.17.2 <https://github.com/SwissDataScienceCenter/renku-ui/releases/tag/3.17.2>`_
-- `renku-graph 2.48.2 <https://github.com/SwissDataScienceCenter/renku-graph/releases/tag/2.48.2>`_ 
+- `renku-graph 2.48.2 <https://github.com/SwissDataScienceCenter/renku-graph/releases/tag/2.48.2>`_
 
 
 0.45.1
 ------
-This is a bugfix release that updates the helm chart to work with new 
-prometheus metrics in the renku core service, which was preventing it from 
+This is a bugfix release that updates the helm chart to work with new
+prometheus metrics in the renku core service, which was preventing it from
 starting properly if metrics were enabled. In addition this release
 also addresses problems with expiring Gitlab access tokens when sessions
 are paused and resumed which caused resumed session to not be able to push to Gitlab
@@ -219,8 +261,8 @@ Internal Changes
 
 **🐞 Bug Fixes**
 
-- **Helm Chart**: update core-service deployment to allow service and rq 
-  metrics to run side-by-side (`#3303 
+- **Helm Chart**: update core-service deployment to allow service and rq
+  metrics to run side-by-side (`#3303
   <https://github.com/SwissDataScienceCenter/renku/pull/3303>`_).
 - **Notebooks**: use a larger /dev/shm folder in sessions
   (`#1723 <https://github.com/SwissDataScienceCenter/renku-notebooks/issues/1723>`_)

RELEASE.md

@@ -14,7 +14,6 @@ This procedure should be followed for *any* release:
 * Create a `CHANGELOG` entry for the release and open a PR; create a deployment, this is the reference for the release.
 * Note that any PR that should go into the release needs to target the release branch _not_ `master`.
 * All release branches should be protected.
-* Use the "Rebase and Merge" button to merge release branches into `master`; do not squash commits.
 
 Acceptance tests have to pass on all release branches before merging. 
 

acceptance-tests/.scalafmt.conf

@@ -1,4 +1,4 @@
-version = "3.7.17"
+version = "3.8.0"
 
 runner.dialect = "scala213"
 

acceptance-tests/README.md

@@ -204,7 +204,6 @@ The test are built using the Page Object Pattern (e.g.
 https://www.pluralsight.com/guides/getting-started-with-page-object-pattern-for-your-selenium-tests) which in
 short is about wrapping an UI page into a class/object and using it in the test script.
 
-
 As mentioned above there's a `target/tests-execution.log` file where tests debug statements from tests execution are written.
 
 ## Project organization

acceptance-tests/build.sbt

@@ -19,7 +19,7 @@
 organization := "ch.renku"
 name := "renku-acceptance-tests"
 version := "0.1.0"
-scalaVersion := "2.13.12"
+scalaVersion := "2.13.13"
 
 Test / parallelExecution := false
 
@@ -30,8 +30,8 @@ publishTo := Some(Resolver.file("Unused transient repository", file("target/unus
 
 val circeVersion = "0.14.6"
 
-libraryDependencies += "ch.qos.logback"          % "logback-classic"               % "1.4.14"
-libraryDependencies += "com.github.pureconfig"  %% "pureconfig"                    % "0.17.5"     % Test
+libraryDependencies += "ch.qos.logback"          % "logback-classic"               % "1.5.3"
+libraryDependencies += "com.github.pureconfig"  %% "pureconfig"                    % "0.17.6"     % Test
 libraryDependencies += "eu.timepit"             %% "refined"                       % "0.11.1"     % Test
 libraryDependencies += "io.circe"               %% "circe-core"                    % circeVersion % Test
 libraryDependencies += "io.circe"               %% "circe-literal"                 % circeVersion % Test
@@ -43,7 +43,7 @@ libraryDependencies += "org.scalacheck"         %% "scalacheck"
 libraryDependencies += "org.scalatest"          %% "scalatest"                     % "3.2.18"     % Test
 libraryDependencies += "org.scalatestplus"      %% "selenium-4-1"                  % "3.2.12.1"   % Test
 libraryDependencies += "org.seleniumhq.selenium" % "selenium-http-jdk-client"      % "4.13.0"     % Test
-libraryDependencies += "org.seleniumhq.selenium" % "selenium-java"                 % "4.17.0"     % Test
+libraryDependencies += "org.seleniumhq.selenium" % "selenium-java"                 % "4.18.1"     % Test
 libraryDependencies += "org.slf4j"               % "slf4j-log4j12"                 % "2.0.11"     % Test
 libraryDependencies += "org.typelevel"          %% "cats-effect"                   % "3.5.3"      % Test
 libraryDependencies += "org.typelevel"          %% "cats-effect-testing-scalatest" % "1.5.0"      % Test

acceptance-tests/project/build.properties

@@ -1 +1 @@
-sbt.version=1.9.8
+sbt.version=1.9.9

chartpress.yaml

@@ -19,6 +19,7 @@ charts:
       - helm-chart
       - acceptance-tests
       - scripts/init-realm
+      - scripts/init-db
     images:
       tests:
         buildArgs:

helm-chart/renku/requirements.yaml

@@ -1,35 +1,34 @@
 dependencies:
-- name: gitlab
-  repository: "https://swissdatasciencecenter.github.io/helm-charts/"
-  version: 0.8.0
-  condition: gitlab.enabled
-- name: postgresql
-  version: 9.1.1
-  repository: "https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami"
-  condition: postgresql.enabled
-- name: keycloakx
-  version: 2.1.0
-  repository: "https://codecentric.github.io/helm-charts"
-  condition: keycloakx.enabled
-- name: redis
-  # bitnami claims that this will always contain a full set of charts - let us pray...
-  # this index was 19MB as of the date of this commit and contained redis 17.4.2
-  repository: "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami"
-  version: 17.4.2
-  condition: redis.install
-- name: renku-jena
-  version: "0.0.23"
-  repository: "https://swissdatasciencecenter.github.io/helm-charts/"
-  alias: jena
-- name: amalthea
-  repository: "https://swissdatasciencecenter.github.io/helm-charts/"
-  version: "0.11.0"
-- name: dlf-chart
-  repository: "https://swissdatasciencecenter.github.io/datashim/"
-  version: "0.3.9-renku-2"
-  condition: notebooks.cloudstorage.s3.installDatashim
-- name: csi-rclone
-  repository: "https://swissdatasciencecenter.github.io/helm-charts/"
-  version: "0.1.6"
-  condition: global.csi-rclone.install
-
+  - name: gitlab
+    repository: "https://swissdatasciencecenter.github.io/helm-charts/"
+    version: 0.8.0
+    condition: gitlab.enabled
+  - name: postgresql
+    version: "14.2.4"
+    repository: "oci://registry-1.docker.io/bitnamicharts"
+    condition: postgresql.enabled
+  - name: keycloakx
+    version: 2.1.0
+    repository: "https://codecentric.github.io/helm-charts"
+    condition: keycloakx.enabled
+  - name: redis
+    # bitnami claims that this will always contain a full set of charts - let us pray...
+    # this index was 19MB as of the date of this commit and contained redis 17.4.2
+    repository: "https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami"
+    version: 17.4.2
+    condition: redis.install
+  - name: renku-jena
+    version: "0.0.23"
+    repository: "https://swissdatasciencecenter.github.io/helm-charts/"
+    alias: jena
+  - name: amalthea
+    repository: "https://swissdatasciencecenter.github.io/helm-charts/"
+    version: "0.11.0"
+  - name: dlf-chart
+    repository: "https://swissdatasciencecenter.github.io/datashim/"
+    version: "0.3.9-renku-2"
+    condition: notebooks.cloudstorage.s3.installDatashim
+  - name: csi-rclone
+    repository: "https://swissdatasciencecenter.github.io/helm-charts/"
+    version: "0.1.7"
+    condition: global.csi-rclone.install

helm-chart/renku/templates/_helpers.tpl

@@ -164,3 +164,7 @@ app.kubernetes.io/managed-by: {{ .Release.Service }}
 {{- define "renku.keycloak.realm" -}}
 {{ .Values.global.keycloak.realm | default "Renku" }}
 {{- end -}}
+
+{{- define "renku.dataService.keycloak.clientId" -}}
+data-service
+{{- end -}}

helm-chart/renku/templates/data-service/deployment.yaml

@@ -58,6 +58,13 @@ spec:
               value: {{ (printf "%s://%s/auth/" (include "renku.http" .) .Values.global.renku.domain) | quote }}
             - name: KEYCLOAK_TOKEN_SIGNATURE_ALGS
               value: "RS256"
+            - name: KEYCLOAK_CLIENT_ID
+              value: {{ include "renku.dataService.keycloak.clientId" . | quote }}
+            - name: KEYCLOAK_CLIENT_SECRET
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "renku.fullname" . }}
+                  key: dataServiceKeycloakClientSecret
             - name: SERVER_DEFAULTS
               value: /etc/renku-data-service/server_options/server_defaults.json
             - name: SERVER_OPTIONS

helm-chart/renku/templates/keycloak-users-sync-cronjob.yaml

@@ -0,0 +1,113 @@
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: {{ include "renku.fullname" . }}-keycloak-sync-events
+  labels:
+    app: keycloak-sync
+    heritage: {{ .Release.Service }}
+    release: {{ .Release.Name }}
+    chart: {{ template "renku.chart" . }}
+spec:
+  schedule: "*/2 * * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      template:
+        metadata:
+          labels:          
+            app: keycloak-sync
+        spec:
+          initContainers:
+            {{- include "certificates.initContainer" . | nindent 12 }}
+          containers:
+            - name: keycloak-sync
+              image: "{{ .Values.dataService.keycloakSync.image.repository }}:{{ .Values.dataService.keycloakSync.image.tag }}"
+              imagePullPolicy: IfNotPresent
+              env:
+                - name: DB_HOST
+                  value: {{ template "postgresql.fullname" . }}
+                - name: DB_USER
+                  value: {{ .Values.global.db.common.username }}
+                - name: DB_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: {{ .Values.global.db.common.passwordSecretName }}
+                      key: password
+                - name: KEYCLOAK_URL
+                  value: {{ include "renku.keycloakUrl" . | quote}}
+                - name: KEYCLOAK_REALM
+                  value: {{ include "renku.keycloak.realm" . | quote}}
+                - name: KEYCLOAK_CLIENT_ID
+                  value: {{ include "renku.dataService.keycloak.clientId" . | quote }}
+                - name: KEYCLOAK_CLIENT_SECRET
+                  valueFrom:
+                    secretKeyRef:
+                      name: {{ include "renku.fullname" . }}
+                      key: dataServiceKeycloakClientSecret
+                - name: TOTAL_USER_SYNC
+                  value: "false"
+                {{- include "certificates.env.python" . | nindent 16 }}
+              volumeMounts:
+                {{- include "certificates.volumeMounts.system" . | nindent 16 }}
+          restartPolicy: Never
+          volumes:
+            {{- include "certificates.volumes" . | nindent 12 }}
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: {{ include "renku.fullname" . }}-keycloak-sync-total
+  labels:
+    app: keycloak-sync
+    heritage: {{ .Release.Service }}
+    release: {{ .Release.Name }}
+    chart: {{ template "renku.chart" . }}
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  successfulJobsHistoryLimit: 1
+  failedJobsHistoryLimit: 3
+  jobTemplate:
+    spec:
+      template:
+        metadata:
+          labels:          
+            app: keycloak-sync
+        spec:
+          initContainers:
+            {{- include "certificates.initContainer" . | nindent 12 }}
+          containers:
+            - name: keycloak-sync
+              image: "{{ .Values.dataService.keycloakSync.image.repository }}:{{ .Values.dataService.keycloakSync.image.tag }}"
+              imagePullPolicy: IfNotPresent
+              env:
+                - name: DB_HOST
+                  value: {{ template "postgresql.fullname" . }}
+                - name: DB_USER
+                  value: {{ .Values.global.db.common.username }}
+                - name: DB_PASSWORD
+                  valueFrom:
+                    secretKeyRef:
+                      name: {{ .Values.global.db.common.passwordSecretName }}
+                      key: password
+                - name: KEYCLOAK_URL
+                  value: {{ include "renku.keycloakUrl" . | quote}}
+                - name: KEYCLOAK_REALM
+                  value: {{ include "renku.keycloak.realm" . | quote}}
+                - name: KEYCLOAK_CLIENT_ID
+                  value: {{ include "renku.dataService.keycloak.clientId" . | quote }}
+                - name: KEYCLOAK_CLIENT_SECRET
+                  valueFrom:
+                    secretKeyRef:
+                      name: {{ include "renku.fullname" . }}
+                      key: dataServiceKeycloakClientSecret
+                - name: TOTAL_USER_SYNC
+                  value: "true"
+                {{- include "certificates.env.python" . | nindent 16 }}
+              volumeMounts:
+                {{- include "certificates.volumeMounts.system" . | nindent 16 }}
+          restartPolicy: Never
+          volumes:
+            {{- include "certificates.volumes" . | nindent 12 }}

helm-chart/renku/templates/network-policies.yaml

@@ -69,6 +69,12 @@ spec:
           namespaceSelector:
             matchLabels:
               kubernetes.io/metadata.name: {{ .Release.Namespace }}
+        - podSelector:
+            matchLabels:
+              app: keycloak-sync
+          namespaceSelector:
+            matchLabels:
+              kubernetes.io/metadata.name: {{ .Release.Namespace }}
       ports:
         - protocol: TCP
           port: 5432