Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge data connectors and notebooks/amalthea migration #428

Conversation

leafty
Copy link
Member

@leafty leafty commented Sep 26, 2024

WIP

Merges:

Currently working:

  • Can create a project with the UI or swagger
  • Can create a data connector with the UI or swagger
  • Can link the two with swagger
  • Can create a session launcher with swagger
  • Can launch the session with swagger
  • Public access data is mounted

/deploy renku=feat-jupyter-free-sessions amalthea-sessions=feat-add-cloud-storage renku-notebooks=master renku-gateway=feat-add-renku-refresh-token-for-data-service renku-ui=build/data-connectors extra-values=amalthea-sessions.deployCrd=false,amalthea-sessions.rcloneStorageClass=csi-rclone-387-secret-annotation #notest

Moves the `project_slugs` table to `entity_slugs` and `project_slugs_old` to `entity_slugs_old`. This is done to prepare for more entity types to be namespaced.

Note: the sequence migration has been tested:
1. Force migration revision to be `9058bf0a1a12` (before this).
2. Deploy with image tag `renku/renku-data-service:sha-345a1bb` (current `main`) and run `alembic check` and `alembic heads` to verify the DB is before this change.
3. Create some projects.
4. Re-deploy this PR and check that migration revision is `a11752a5afba` (after this).
5. Create more projects.
Introduce namespaced data connectors for Renku 2.0.

Details:
* Remove API routes for `/storages_v2`
* Add API routes for `/data_connectors`
* Add `data_connectors` table
* Drop the `storage_secrets` table
* Add `data_connectors_secrets` table
Fixes deleting storage configuration fields by setting them to `null`.
…to leafty/feat-namespaced-data-connectors-3

# Conflicts:
#	test/bases/renku_data_services/data_api/test_data_connectors.py
…ectors-3' into leafty/feat-namespaced-data-connectors-4
…ectors-4' into leafty/feat-namespaced-data-connectors-5
…ectors-4' into leafty/feat-namespaced-data-connectors-5
@RenkuBot
Copy link
Contributor

You can access the deployment of this PR at https://renku-ci-ds-428.dev.renku.ch

@olevski olevski force-pushed the feat-add-cloudstorage branch 2 times, most recently from bf8efeb to 04553ec Compare September 27, 2024 13:03
Base automatically changed from feat-add-cloudstorage to release-amaltheas-migration September 27, 2024 14:01
…o leafty/feat-namespaced-data-connectors-merge-amalthea-migration

# Conflicts:
#	Makefile
#	bases/renku_data_services/data_api/app.py
#	components/renku_data_services/notebooks/api.spec.yaml
#	components/renku_data_services/notebooks/api/classes/k8s_client.py
#	components/renku_data_services/notebooks/api/classes/server.py
#	components/renku_data_services/notebooks/api/classes/server_manifest.py
#	components/renku_data_services/notebooks/api/schemas/servers_get.py
#	components/renku_data_services/notebooks/apispec.py
#	components/renku_data_services/notebooks/blueprints.py
#	components/renku_data_services/session/api.spec.yaml
#	components/renku_data_services/storage/blueprints.py
#	components/renku_data_services/storage/models.py
#	poetry.lock
#	projects/renku_data_service/pyproject.toml
#	projects/secrets_storage/pyproject.toml
#	test/bases/renku_data_services/data_api/test_sessions.py
…o leafty/feat-namespaced-data-connectors-merge-amalthea-migration
@leafty leafty temporarily deployed to renku-ci-ds-428 October 2, 2024 06:50 — with GitHub Actions Inactive
@olevski olevski force-pushed the release-amaltheas-migration branch from 7fcfa17 to be5272b Compare October 2, 2024 13:09
@olevski olevski force-pushed the release-amaltheas-migration branch 5 times, most recently from d2b2d48 to a932966 Compare October 8, 2024 13:59
@olevski olevski force-pushed the release-amaltheas-migration branch 2 times, most recently from b2532b4 to 62e197a Compare October 28, 2024 13:01
@olevski olevski force-pushed the release-amaltheas-migration branch from 91b8e88 to ea89929 Compare October 30, 2024 17:19
@leafty leafty closed this Oct 31, 2024
@leafty leafty deleted the leafty/feat-namespaced-data-connectors-merge-amalthea-migration branch October 31, 2024 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do not merge A PR that has been approved but should not be merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants