Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Python 3.11 #69

Merged
merged 4 commits into from
Jan 16, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,10 @@ condacolab.check()

The default `condacolab.install()` provides Mambaforge, but there are other `conda` distributions to choose from:

- `install_miniconda()`: This will install the Miniconda distribution, using a version built for Python 3.10. Unlike Anaconda, this distribution only contains `python` and `conda`.
- `install_miniconda()`: This will install the Miniconda distribution, using a version built for Python 3.11. Unlike Anaconda, this distribution only contains `python` and `conda`.
- `install_miniforge()`: Like Miniconda, but built off `conda-forge` packages. The Miniforge distribution is officially provided by [conda-forge](https://github.com/conda-forge/miniforge).
- `install_mambaforge()`: Like Miniforge, but with `mamba` included. The Mambaforge distribution is officially provided by [conda-forge](https://github.com/conda-forge/miniforge).

For advanced users, `install_from_url()` is also available. It expects a URL pointing to a [`constructor`-like installer](https://github.com/conda/constructor), so you can prebuild a Python 3.10 distribution that fulfills your own needs.
For advanced users, `install_from_url()` is also available. It expects a URL pointing to a [`constructor`-like installer](https://github.com/conda/constructor), so you can prebuild a Python 3.11 distribution that fulfills your own needs.

> If you want to build your own `constructor`-based installer, check the FAQ below!

Expand Down Expand Up @@ -67,7 +66,7 @@ If you have a environment file (e.g. `environment.yml`), you can use it like thi

### How does it work?

Google Colab runs on Python 3.10. We install the Miniconda distribution on top of the existing one at `/usr/local`, add a few configuration files so we stay with Python 3.10 and the newly installed packages are available. Finally, we wrap the Python executable to redirect and inject some environment variables needed to load the new libraries. Since we need to re-read `LD_LIBRARY_PATH`, a kernel restart is needed.
Google Colab runs on Python 3.11. We install the Miniconda distribution on top of the existing one at `/usr/local`, add a few configuration files so we stay with Python 3.11 and the newly installed packages are available. Finally, we wrap the Python executable to redirect and inject some environment variables needed to load the new libraries. Since we need to re-read `LD_LIBRARY_PATH`, a kernel restart is needed.

### How can I cache my installation? I don't want to wait every time I start Colab.

Expand Down
117 changes: 62 additions & 55 deletions condacolab.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
raise RuntimeError("This module must ONLY run as part of a Colab notebook!")


__version__ = "0.1.9"
__version__ = "0.1.10"
__author__ = "Jaime Rodríguez-Guerra <[email protected]>"


Expand All @@ -39,7 +39,7 @@
def _chunked_sha256(path, chunksize=1_048_576):
hasher = hashlib.sha256()
with open(path, "rb") as f:
while (chunk := f.read(chunksize)):
while chunk := f.read(chunksize):
hasher.update(chunk)
return hasher.hexdigest()

Expand Down Expand Up @@ -95,7 +95,9 @@ def install_from_url(

if sha256 is not None:
digest = _chunked_sha256(installer_fn)
assert digest == sha256, f"💥💔💥 Checksum failed! Expected {sha256}, got {digest}"
assert (
digest == sha256
), f"💥💔💥 Checksum failed! Expected {sha256}, got {digest}"

print("📦 Installing...")
task = run(
Expand Down Expand Up @@ -167,15 +169,16 @@ def install_from_url(
get_ipython().kernel.do_shutdown(True)


def install_mambaforge(
prefix: os.PathLike = PREFIX, env: Dict[AnyStr, AnyStr] = None, run_checks: bool = True
def install_miniforge(
prefix: os.PathLike = PREFIX,
env: Dict[AnyStr, AnyStr] = None,
run_checks: bool = True,
):
"""
Install Mambaforge 23.11.0, built for Python 3.10.
Install Miniforge 24.11.1, built for Python 3.11.

Mambaforge consists of a Miniconda-like distribution optimized
and preconfigured for conda-forge packages, and includes ``mamba``,
a faster ``conda`` implementation.
Miniforge consists of a Miniconda-like distribution optimized
and preconfigured for conda-forge packages.

Parameters
----------
Expand All @@ -196,53 +199,35 @@ def install_mambaforge(
Change to False to ignore checks and always attempt
to run the installation.
"""
installer_url = "https://github.com/conda-forge/miniforge/releases/download/23.11.0-0/Mambaforge-23.11.0-0-Linux-x86_64.sh"
checksum = "3dfdcc162bf0df83b5025608dc2acdbbc575bd416b75701fb5863343c0517a78"
install_from_url(installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum)
installer_url = (
"https://github.com/jaimergp/miniforge/releases/download/"
"24.11.2-1_colab/Miniforge3-colab-24.11.2-1_colab-Linux-x86_64.sh"
)
checksum = "359f81e4bc706c3e237105cc42cb5d5deaee0c89b4391cb70c5cb2c27d4e3677"
install_from_url(
installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum
)


# Make mambaforge the default
install = install_mambaforge


def install_miniforge(
prefix: os.PathLike = PREFIX, env: Dict[AnyStr, AnyStr] = None, run_checks: bool = True
):
"""
Install Miniforge 23.11.0, built for Python 3.10.

Miniforge consists of a Miniconda-like distribution optimized
and preconfigured for conda-forge packages.
install = install_miniforge

Parameters
----------
prefix
Target location for the installation
env
Environment variables to inject in the kernel restart.
We *need* to inject ``LD_LIBRARY_PATH`` so ``{PREFIX}/lib``
is first, but you can also add more if you need it. Take
into account that no quote handling is done, so you need
to add those yourself in the raw string. They will
end up added to a line like ``exec env VAR=VALUE python3...``.
For example, a value with spaces should be passed as::

env={"VAR": '"a value with spaces"'}
run_checks
Run checks to see if installation was run previously.
Change to False to ignore checks and always attempt
to run the installation.
"""
installer_url = "https://github.com/conda-forge/miniforge/releases/download/23.11.0-0/Miniforge3-23.11.0-0-Linux-x86_64.sh"
checksum = "73576b96409ed38a7ca596bece058e8c77c6ef3eab42af7cfdf2ae975e8f3928"
install_from_url(installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum)
def install_mambaforge(*args, **kwargs):
print(
"Mambaforge has been sunset. It is now identical to Miniforge. Installing Miniforge...",
file=sys.stderr,
)
install_miniforge(*args, **kwargs)


def install_miniconda(
prefix: os.PathLike = PREFIX, env: Dict[AnyStr, AnyStr] = None, run_checks: bool = True
prefix: os.PathLike = PREFIX,
env: Dict[AnyStr, AnyStr] = None,
run_checks: bool = True,
):
"""
Install Miniconda 23.11.0 for Python 3.10.
Install Miniconda 24.11.1 for Python 3.11.

Parameters
----------
Expand All @@ -263,17 +248,28 @@ def install_miniconda(
Change to False to ignore checks and always attempt
to run the installation.
"""
installer_url = "https://repo.anaconda.com/miniconda/Miniconda3-py310_23.11.0-2-Linux-x86_64.sh"
installer_url = (
"https://repo.anaconda.com/miniconda/Miniconda3-py311_24.11.1-0-Linux-x86_64.sh"
)
checksum = "35a58b8961e1187e7311b979968662c6223e86e1451191bed2e67a72b6bd0658"
install_from_url(installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum)
print(
"Miniconda is subject to terms of service:",
"https://legal.anaconda.com/policies/en/#terms-of-service",
file=sys.stderr,
)
install_from_url(
installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum
)


def install_anaconda(
prefix: os.PathLike = PREFIX, env: Dict[AnyStr, AnyStr] = None, run_checks: bool = True
prefix: os.PathLike = PREFIX,
env: Dict[AnyStr, AnyStr] = None,
run_checks: bool = True,
):
"""
Install Anaconda 2023.03, the latest version built
for Python 3.10 at the time of update.
Install Anaconda 2024.02, the latest version built
for Python 3.11 at the time of update.

Parameters
----------
Expand All @@ -294,9 +290,18 @@ def install_anaconda(
Change to False to ignore checks and always attempt
to run the installation.
"""
installer_url = "https://repo.anaconda.com/archive/Anaconda3-2023.03-1-Linux-x86_64.sh"
checksum = "95102d7c732411f1458a20bdf47e4c1b0b6c8a21a2edfe4052ca370aaae57bab"
install_from_url(installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum)
installer_url = (
"https://repo.anaconda.com/archive/Anaconda3-2024.02-1-Linux-x86_64.sh"
)
checksum = "c536ddb7b4ba738bddbd4e581b29308cb332fa12ae3fa2cd66814bd735dff231"
print(
"Anaconda Distribution is subject to terms of service:",
"https://legal.anaconda.com/policies/en/#terms-of-service",
file=sys.stderr,
)
install_from_url(
installer_url, prefix=prefix, env=env, run_checks=run_checks, sha256=checksum
)


def check(prefix: os.PathLike = PREFIX, verbose: bool = True):
Expand All @@ -316,7 +321,9 @@ def check(prefix: os.PathLike = PREFIX, verbose: bool = True):

pymaj, pymin = sys.version_info[:2]
sitepackages = f"{prefix}/lib/python{pymaj}.{pymin}/site-packages"
assert sitepackages in sys.path, f"💥💔💥 PYTHONPATH was not patched! Value: {sys.path}"
assert (
sitepackages in sys.path
), f"💥💔💥 PYTHONPATH was not patched! Value: {sys.path}"
assert (
f"{prefix}/bin" in os.environ["PATH"]
), f"💥💔💥 PATH was not patched! Value: {os.environ['PATH']}"
Expand Down
8 changes: 4 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
[tool.poetry]
name = "condacolab"
version = "0.1.9"
version = "0.1.10"
description = "Install Conda and friends on Google Colab, easily"
readme = "README.md"
authors = ["Jaime Rodríguez-Guerra <[email protected]>"]
license = "MIT"
homepage = "https://github.com/jaimergp/condacolab"
repository = "https://github.com/jaimergp/condacolab"
homepage = "https://github.com/conda-incubator/condacolab"
repository = "https://github.com/conda-incubator/condacolab"

[tool.poetry.dependencies]
python = "^3.10"
python = "^3.11"

[build-system]
requires = ["poetry-core>=1.0.0"]
Expand Down