-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ShapeUp] Renku 2.0 Sessions #3475
Comments
This was referenced Feb 7, 2024
Closed
This was referenced Feb 14, 2024
This was referenced Feb 22, 2024
Closed
leafty
added a commit
to SwissDataScienceCenter/renku-data-services
that referenced
this issue
Mar 14, 2024
See SwissDataScienceCenter/renku#3475. * Adds API endpoints for session environments * Adds API endpoints for session launchers
leafty
added a commit
to SwissDataScienceCenter/renku-notebooks
that referenced
this issue
Mar 19, 2024
Fixes #1800. See SwissDataScienceCenter/renku#3475. Add support for Renku 2.0 sessions: * Add a new endpoint `/v2/servers` for starting Renku 2.0 sessions * Renku 2.0 sessions can have any number of code repositories mounted from GitLab * Add new session annotation to describe Renku 2.0 sessions Co-authored-by: M. Alisafaee <[email protected]>
leafty
added a commit
to SwissDataScienceCenter/renku-ui
that referenced
this issue
Mar 25, 2024
Add initial support for Renku 2.0 sessions. * Add new features for sessions in Renku 2.0: session environments and session launchers * Add session environment management to admin page * Add session launchers to Renku 2.0 projects * Can start a session from a session launcher * Can open a running session * Can perform session actions (pause, delete, modify) * Renku 1.0 and Renku 2.0 sessions are not co-mingled: Renku 1.0 sessions continue to appear in the dashboard, etc. while Renku 2.0 sessions will appear only in `/v2` pages. Limitations: * Sessions can work with projects with zero repositories. * Sessions for projects with at least one repository will only work if the first repository is in GitLab. See SwissDataScienceCenter/renku#3475.
Close by #3569. |
olevski
pushed a commit
to SwissDataScienceCenter/renku-data-services
that referenced
this issue
Jul 10, 2024
Fixes #1800. See SwissDataScienceCenter/renku#3475. Add support for Renku 2.0 sessions: * Add a new endpoint `/v2/servers` for starting Renku 2.0 sessions * Renku 2.0 sessions can have any number of code repositories mounted from GitLab * Add new session annotation to describe Renku 2.0 sessions Co-authored-by: M. Alisafaee <[email protected]>
olevski
added a commit
to SwissDataScienceCenter/renku-data-services
that referenced
this issue
Jul 16, 2024
* style: fix code style and add tests * refactor: increase encapsulation; minor bug fixes * feat: spawn new server for different branch * use Kubernetes api for notebook queries (#185) * refactor: minor formatting and unnecessary code removal * feat: include notebook username in pod labels * refactor: minor * feat: use k8s API to get user Jupyter servers * refactor: remove ununsed code * fix: minor fix in README.rst * feat: return server data as needed by UI Co-authored-by: rokroskar <[email protected]> * build: make tests pass * fix: apply review comments * fix: uncaught exception when creating notebooks * fix: use local and remote shortened commit-sha to recover from autosaves (#190) * fix: use shortened commit-sha to check for an autosave match * fix: use commit-sha of remote branch to create autosave branch name * fix: avoid a None dereferencing * fix: use both local and remote commit-sha for autosave * feat: implement new notebook API * feat: can force delete pods (#205) * refactor: add more logs (#207) * restore telepresence script (#209) fix: update telepresence script and add vs code debugger * feat: use project name in the notebook URL (#211) * feat: handle k8s api errors when querying for logs (#216) * feat: handle k8s api errors when querying for logs (#216) * feat: update server_logs api (#225) feat: return json data from server_logs and expose lines limit fix #224 * chore: update werkzeug behavior (#242) * feat: add requested resources info to servers endpoint (#261) fix #223 * fix: use registry API to find image closes #273 * feat: allow unprivileged an anonymous users to launch servers - Enable the use of tmpauthenticator for Jupyterhub authentication. This enables notebook sessions for logged-out users. Closes #171 - Enable logged in users without write access to a project to start notebook servers from the project. Closes #275 - Add a helper script which creates a helm values file for a secondary notebooks service deployment from the main Renku values file (some amount of guessing involved). * fix: use lowercase image repository names closes #287 * feat: use user credentials for pulling images (#327) * fix(app): escape username in registry secret name * chore: add sentry environment (#434) * feat: automatically remove user registry secrets (#435) Add cronjob that periodically deletes users image pull secrets along with unit tests * bug: fix cull secrets failures with pods forked repos * feat: specify image in server request * feat(app): move gitlab oauth token to separate proxy container * bug: launching sessions that are part of a subgroup fails (#493) * fix: cannot create user env due to invalid k8s secret labels * fix: properly recognize pod status when terminating (#497) * fix: properly determine user server pod status with multiple containers Co-authored-by: Andreas Bleuler <[email protected]> * refactor: define models and schemas (#480) * chore: simplify tests * chore: make JH start once per session * feat: add schemas and swagger * feat: add swagger descriptions and security schema * chore: add docstrings to schema classes * fix: return server details when pending and spawning to prevent UI content flicker * fix: remove unneccessary log error calls (#512) * fix: pod resources schema (#513) * fix: add missing fields in renku-notebooks schemas * chore: update tests * fix: simplify definition of UserPodResources schema * fix: retry when creating notebook fails with 500 (#518) * fix: retry when creating notebook fails with 500 * fix: capture protocolerror in spawner, log and reraise * fix: add 1 sec sleep when retrying failed server launch * feat: validate server options (#529) * feat: add jsonschema validation for server options in values.yaml * feat: add specific schemas for server options in notebooks api * fix: validate server options values are valid when launching notebook Co-authored-by: Andreas Bleuler <[email protected]> * fix: unknown gpu field in resources request (#538) * fix: properly handle gpu field in resources request * fix: do not check defaultUrl against list of server options (#542) * fix: allow any values in default url server options, not just fixed choices * fix: allow and include unknown annotations in schema (#546) * fix: labels values are too long for k8s (#553) * fix: labels values are too long for k8s * fix: use project id as label instead of truncated projectname * feat: use pvs for user sessions (#573) * refactor: use python classes (#541) * enhancement: add server class * enhancement: add user class * enhancement: keep track of k8s pods during testing * refactor: remove response validation, clean up schemas and unused code * chore: move user pvc logic into server class * fix: revert rules.toml to original state * chore: add/revise docstrings * fix: increase huboauth caching from 1min to 5min (#603) * feat: endpoint for autosave information (#607) * feat: add autosave information endpoint * chore: add unit tests for autosave functionality * fix missing annotations handling in marshmallow (#637) * feat: cleanup PVS and autosave branches (#611) * feat: remove old pvcs and autosave branches * fix: remove disk request values (#657) * fix: remove disk request option from values.yaml * fix: add default value to disk request in server options to be used when PVCs are enabled and disk requests in the server options are disabled * fix: properly assign default server option values (#649) * fix: limit emptydir size * fix: properly assign default values to server options * chore: add tests for checking proper server options defaults are used * Add autosave delete endpoint (#673) * feat: add autosave classes * feat: add autosave DELETE endpoint * feat: add autosave names to GET endpoint * chore: dependabot Updates Batch (#705) * chore: bump pytest-cov from 2.12.0 to 2.12.1 Bumps [pytest-cov](https://github.com/pytest-dev/pytest-cov) from 2.12.0 to 2.12.1. - [Release notes](https://github.com/pytest-dev/pytest-cov/releases) - [Changelog](https://github.com/pytest-dev/pytest-cov/blob/master/CHANGELOG.rst) - [Commits](pytest-dev/pytest-cov@v2.12.0...v2.12.1) * fix: properly display storage (#708) * fix: do not use ephemeral storage in server responses - use proper storage numbers * fix: use same format for memory and storage * fix: use username in pvc name * chore: fix telepresence script Co-authored-by: Rok Roškar <[email protected]> * fix: upper case letters in pvc name (#713) * fix: upper case letters in pvc name * fix: use escapism on project name used in k8s pvc name * fix: failing on pvcs created with old 0.8.4 renku verison pvc naming (#715) * fix: listing sessions when a project has been deleted (#718) * fix: handle missing git resources gracefully when listing sessions * fix: do not raise errors when validating autosaves * chore: commit latest version of requirements.lock file * fix: handle invalid pvc based autosaves * fix: bugs with mistyped variable and missing branches in autosaves (#723) * feat: use Amalthea to run sessions through a K8s operator (#668) - refactoring of the notebook service - adapt existing and expand test suite Co-authored-by: Andreas Bleuler <[email protected]> * fix: use `jupyter notebook` as arg for launching a server * chore: prep release 1.0.0 (#755) * chore: update ingress to v1 * chore: update readme for amalthea * chore: update changelog for 1.0.0 release * fix(tests): pin setuptools version in test image build Co-authored-by: Andreas Bleuler <[email protected]> Co-authored-by: Rok Roškar <[email protected]> * fix(app): when a gitlab project does not exist log a warning instead of error (#763) * fix(app): use the right storage class name parameter in Jupyter server manifest (#769) * feat(app): option to not limit size of user session emptyDir (#785) * fix(app): displaying gpu resources in API responses (#786) * fix: add ingress annotation to each session so iframes work (#794) * fix(app): check out the correct branch instead of always master(#802) * feat(chart): add session tolerations, affinity, nodeSelector (#806) * fix(app): do not delete autosaves before restoring (#814) * fix(app): use pyjwt to decode token (#818) Co-authored-by: Andreas Bleuler <[email protected]> * fix(app): apply CPU limits to user sessions (#826) * fix(app): use 3 scenarios for session cpu limits (#828) * fix(app): user not authenticated and anonymous sessions not allowed (#829) * fix(app): return proper values for image and repository (#834) * fix(app): do not accept an empty anonymous user id (#845) * feat(app): enable age-based session culling (#848) * fix(app): case insensitive project and namespace (#858) enhancement(app): add the ability to do simple migrations of the jupyterserver k8s resources * feat: mount s3 buckets in sessions (#729) Co-authored-by: Andreas Bleuler <[email protected]> * feat: modify for custom CA certificates (#788) * chore: fix swagger page (#894) Co-authored-by: Rok Roškar <[email protected]> * chore(app): enable sentry performance monitoring (#907) * fix(app)!: change s3 bucket naming (#922) * build: update sentry values (#912) * chore: Dependabot updates batch (#917) * chore: bump black from 21.7b0 to 21.9b0 (#746) Bumps [black](https://github.com/psf/black) from 21.7b0 to 21.9b0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](https://github.com/psf/black/commits) * fix(app): remove S3 flag from server options (#940) * feat(app): version endpoint (#938) * feat(app)!: better messages on session launch fail (#923) BREAKING CHANGE: The status of the session is reported in a different more compact form in the API. This affects the `/servers` or `/servers/<server_name>` endpoints. * feat(app): return logs from all containers (#887) Co-authored-by: Ralf Grubenmann <[email protected]> * fix(app): enable s3 flag in server options (#969) * feat(app): use python for git clone (#956) Co-authored-by: Ralf Grubenmann <[email protected]> * feat(app): add sentry to git clone (#963) Co-authored-by: Ralf Grubenmann <[email protected]> * feat(app): add endpoint to verify docker image availability (#990) Co-authored-by: Tasko Olevski <[email protected]> * fix(app): keep git-proxy alive on session shutdown (#951) Co-authored-by: Viktor Gal <[email protected]> * chore: run as non-root (#1020) * chore: set runAsNonRoot to true (#1031) * fix(app): eliminate unnecessary gitlab calls (#1035) * feat: add git error codes to notebooks response (#1055) * squashme: make error messages properly surface (#1064) Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Tasko Olevski <[email protected]> * enahance(app): minimize gitlab api calls (#1087) * enahance(app): minimize gitlab api calls * chore: fix unhandled errors * chore: bump pyjwt from 1.7.1 to 2.4.0 (#1079) * chore: bump pyjwt from 1.7.1 to 2.4.0 Bumps [pyjwt](https://github.com/jpadilla/pyjwt) from 1.7.1 to 2.4.0. - [Release notes](https://github.com/jpadilla/pyjwt/releases) - [Changelog](https://github.com/jpadilla/pyjwt/blob/master/CHANGELOG.rst) - [Commits](jpadilla/pyjwt@1.7.1...2.4.0) * fix(app): s3 bucket location (#1083) * fix(app): check bucket location is in endpoint * fix(app): accept only valid urls for s3 endpoints Co-authored-by: Rok Roškar <[email protected]> * feat(app): surface message and status when pod is unschedulable (#1088) * chore(app): use state from amalthea * feat: set status to failed when unschedulable * feat(app): show session resource usage (#968) Co-authored-by: Lorenzo Cavazzi <[email protected]> * fix(app): show starting message (#1099) * fix(app): handle missing project in autosave list (#1090) * fix(app): custom certs env variables in git clone (#1101) * fix(app): serialize cpu and gpu request to number (#1103) * fix(app): no main pod in status when getting logs (#1104) * fix(app): properly deserialize server options (#1108) * fix(app): cloudstorage in server_options schema (#1113) * fix(app): parse unschedulable resources message (#1114) * fix(app): list autosaves only for the current user (#1118) * fix(app): handle endpoints for s3 not on AWS (#1124) * feat: add support for passing env variables to notebooks (#1131) Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Tasko Olevski <[email protected]> * chore(app): fix for updating gitlab sdk to 3.6.0 (#1169) * fix(app): listing autosaves and debug statements (#1168) * fix(app): show latest unschedulable message (#1161) * chore(app): document max lines for getting logs (#1184) * feat(app): use dataconf and telepresence (#1186) * chore: use 'sub' JWT field for user id (#1199) * enhancement(app): standardized error responses (#849) Co-authored-by: Andreas Bleuler <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> * fix(app): usernames starting with numbers/symbols (#1213) Co-authored-by: Ralf Grubenmann <[email protected]> * feat(app): add auth and expose rpc server sidecar (#762) Co-authored-by: Rok Roškar <[email protected]> * feat(sidecar): run renku cli commands in rpc server (#1219) Co-authored-by: Rok Roškar <[email protected]> * feat(app): add session url env var (#1231) * feat(sidecar): symlink cloudstorage into project (#1230) * chore(app): reduce image size (#1239) * feat(app): detailed status messages (#1289) * feat(app): detailed status messages * feat(app): update amalthea to 0.6.0 * feat(app): add jupyter server caching and multiple namespace support (#1323) Co-authored-by: Sean Murphy <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> * fix: accept zero as max_lines parameter for logs (#1336) * fix: handle Gitlab token expiry (#1338) * fix(app): snyk vulnerabilities (#1341) * fix(app): propagate environment variables for R (#1339) * feat(app): experimental azure blob storage support (#1374) * feat(session): fail when injecting env vars that exist in session (#1396) Co-authored-by: Tasko Olevski <[email protected]> * feat: ssh access to sessions with jump host (#1389) * fix(app): show if ssh enabled in /version (#1407) * fix(app): missing return statement (#1412) * fix(app): server filtering parameter names (#1423) * feat(app): use crc service (#1483) * chore: update python dependencies (#1557) * feat: add session warnings to jupyter server API responses (#1482) Co-authored-by: Tasko Olevski <[email protected]> * fix(app): parse old server options in request (#1570) * chore: add resource quota label (#1613) * feat(app): use PV to hibernate/resume sessions (#1518) Co-authored-by: Johann-Michael Thiebaut <[email protected]> Co-authored-by: Tasko Olevski <[email protected]> * feat(app): use tolerations and affinities from CRC (#1626) * fix(app): fix session hibernation (#1645) * feat: add support for new cloud storage (#1614) Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Tasko Olevski <[email protected]> * fix secret order and mount path check (#1666) * fix: overwrite existing files with storage mounts (#1672) * fix(app): handle OCI image manifests (#1673) * chore: do not set oauth2proxy version in notebooks (#1644) * fix: scientific notation integer parsing in config (#1662) * fix: properly patch tolerations and affinities (#1695) * fix: patch in a larger value for /dev/shm (#1723) * fix: inject new set of tokens when the session is resumed (#1734) * fix: handle missing gitlab tokens (#1735) * feat: rclone csi support (#1707) * feat: rclone csi support * return cloud storage in responses * add support for virtual switch provider * support readonly mode * switch to new deploy action --------- Co-authored-by: Ralf Grubenmann <[email protected]> * feat: allow patching of resource class (#1728) Co-authored-by: Ralf Grubenmann <[email protected]> * feat: support for renku 2 sessions (#1810) (#1815) Fixes #1800. See SwissDataScienceCenter/renku#3475. Add support for Renku 2.0 sessions: * Add a new endpoint `/v2/servers` for starting Renku 2.0 sessions * Renku 2.0 sessions can have any number of code repositories mounted from GitLab * Add new session annotation to describe Renku 2.0 sessions Co-authored-by: M. Alisafaee <[email protected]> * chore: combined dependency update (#1825) * chore: bump cryptography from 41.0.3 to 41.0.6 in /git_services Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.3 to 41.0.6. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@41.0.3...41.0.6) * fix: project path case sensitivity (#1857) * fix: make dataconf ignore unexpected env vars (#1866) * feat: add sidecar for mounting secrets (#1847) * add sidecar for mounting secrets * enable ruff in ci * feat: implement secrets handling (#1853) * feature(server): add support for user secrets The secrets are the ones the user are storing in the secrets storage service. The notebook service will request the creation of a k8s secret from the service that will contain all the secrets to be used in the session. * feature: add support for user secrets in the API A new optional field named user_secrets has been added to the server creation endpoint that will trigger the k8s secret creation upon successful JupyterServer creation. This allows to tie the secret to the JupyterServer object and thus automate its deletion upon jupyter server deletion. --------- Co-authored-by: Samuel Gaist <[email protected]> * feat: support connected services (#1874) Add support to mount repositories from external sources in session. This requires `connected services` support from `renku-data-services`. Also, the API for starting Renku 2.0 sessions has been modified to accept external repositories. Changes in `notebooks`: * Refactored the `UserServer` class to be an abstract class. `Renku1UserServer` and `Renku2UserServer` inherit from `UserServer` and contain the corresponding adaptations needed to support Renku 1.0 and Renku 2.0 sessions respectively. * Changed the `amalthea` patches to reflect changes in `git-clone` and `git-proxy`. * Updated the `patch_statefulset_tokens()` method invoked when resuming sessions. Changes in `git-clone`: * Refactored the `git-clone` container to support cloning repositories from any source. * Cloning is done optimistically. The `git-clone` container will not crash if a repository cannot be cloned. * For private repositories, a `git provider` must be configured and will be used to clone. * The container now only uses the `renku_access_token`. Changes in `git-proxy`: * Refactored the `git-proxy` container to support injecting credentials from external services. * The `git-proxy` is a simple pass-through for anonymous sessions (meaning it should probably not run for anonymous sessions). * Repositories matching a configured `git provider` will have credentials injected: - From `renku-gateway-auth` for the internal GitLab - From `renku-data-services` for other services * The container now only uses the `renku_access_token` and the `renku_refresh_token`. * fix: do not setup any git provider for anonymous sessions (#1879) Fixes #1878. * feat: restrict secrets mount paths (#1873) * fix: remove subpath "None" from sidecar container for Renku 2.0 sessions (#1880) * feat: custom culling times (#1852) * fix: do not add storage mounts patches when a session has no repository (#1892) * chore: add kubernetes_asyncio * refactor: split the service into components closes #145 closes #109 * squashme: move http handlers to blueprints * feat: move notebooks code to data service --------- Co-authored-by: Rok Roškar <[email protected]> Co-authored-by: Mohammad Alisafaee <[email protected]> Co-authored-by: Mohammad Alisafaee <[email protected]> Co-authored-by: Lorenzo Cavazzi <[email protected]> Co-authored-by: Rok Roskar <[email protected]> Co-authored-by: Andreas Bleuler <[email protected]> Co-authored-by: Andreas Bleuler <[email protected]> Co-authored-by: Rok Roškar <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Viktor Gal <[email protected]> Co-authored-by: Ralf Grubenmann <[email protected]> Co-authored-by: Sean Murphy <[email protected]> Co-authored-by: Johann-Michael Thiebaut <[email protected]> Co-authored-by: Flora Thiebaut <[email protected]> Co-authored-by: Samuel Gaist <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Pitch on Notion
The text was updated successfully, but these errors were encountered: