Fix tox + uv caching #29

dfsnow · 2024-12-02T19:24:25Z

This PR updates our GitHub Actions workflows to fix the caching integration between tox and uv. It also switches most workflows over to using uv's built-in Python setup rather than the setup-python GitHub Action.

dfsnow · 2024-12-02T22:14:33Z

.github/workflows/lint.yaml

+        shell: bash
+        run: uv venv


This is actually faster than using setup-python and isolates things in a virtual env. One downside is that it just installs the latest Python version by default (3.13 currently). If we want to pin a version we could add one to this command or add a .python-version file to the root of our project.

[Praise] I like this! Nice that it's faster, and I also think it's helpful that it more closely matches the way we use uv for development.

dfsnow · 2024-12-02T22:14:56Z

.github/workflows/pages.yaml

      - name: Install uv
-        uses: astral-sh/setup-uv@v3
-        with:
-          enable-cache: true
-          cache-dependency-glob: pyproject.toml
-          cache-suffix: docs
+        uses: astral-sh/setup-uv@v4


It's actually faster to run this without using the cache.

[Praise] Common uv W 🚀

dfsnow · 2024-12-02T22:16:24Z

.github/workflows/pypi-publish.yaml

-        uses: astral-sh/setup-uv@v3
-        with:
-          enable-cache: true
-          cache-dependency-glob: pyproject.toml
-
-      - name: Setup Python
-        uses: actions/setup-python@v5
-        with:
-          python-version-file: pyproject.toml
-
-      - name: Install Python dependencies
-        run: uv pip install .


None of this stuff is actually necessary to build the package, so we can remove it.

dfsnow · 2024-12-02T22:26:10Z

.github/workflows/python-build-and-test.yaml

        with:
          enable-cache: true
          cache-dependency-glob: pyproject.toml
-          cache-suffix: tox
+          cache-suffix: ${{ matrix.python-version }}-tox


This is the cause of the caching issue identified in ccao-data/ccao#33. Basically, uv disables caching in CI by default because it's slower to recover the cache than it is to just uv pip install (wild).

It's supposed to cache built-from-source wheels even when pruning is enabled. However, there's a race condition in the testing setup since the first-to-finish jobs are the ones to populate the cache, and those jobs won't have the built-from-source wheel.

The easy fix here is to just give each Python version its own cache.

[Praise] This is wild 🤯 Great find!

dfsnow · 2024-12-02T22:26:25Z

pyproject.toml

+passenv =
+    UV_CACHE_DIR
+    PYTHONUNBUFFERED


Not strictly necessary I don't think, but also can't hurt.

jeancochrane

Awesome work! I'll add these learnings over to my PR on ccao too.

jeancochrane · 2024-12-02T22:47:16Z

.github/workflows/lint.yaml

+        shell: bash
+        run: uv venv


[Praise] I like this! Nice that it's faster, and I also think it's helpful that it more closely matches the way we use uv for development.

jeancochrane · 2024-12-02T22:48:31Z

.github/workflows/lint.yaml


      - name: Install dependencies
        run: uv pip install . ruff

      - name: Lint with ruff
-        run: ruff check --output-format=github .
+        run: uv run ruff check --output-format=github .


[Question, non-blocking] This is totally unrelated to this PR, so no need to resolve now, but I wonder if we really need this lint workflow at all given that pre-commit is running ruff anyway?

Good thinking, another holdover from the R package. I removed it in 55e3225.

jeancochrane · 2024-12-02T22:48:57Z

.github/workflows/pages.yaml

      - name: Install uv
-        uses: astral-sh/setup-uv@v3
-        with:
-          enable-cache: true
-          cache-dependency-glob: pyproject.toml
-          cache-suffix: docs
+        uses: astral-sh/setup-uv@v4


[Praise] Common uv W 🚀

jeancochrane · 2024-12-02T22:50:28Z

.github/workflows/python-build-and-test.yaml

        with:
          enable-cache: true
          cache-dependency-glob: pyproject.toml
-          cache-suffix: tox
+          cache-suffix: ${{ matrix.python-version }}-tox


[Praise] This is wild 🤯 Great find!

Split tox setup and run

0a87824

dfsnow force-pushed the dfsnow/setup-tox-caching branch 3 times, most recently from aca597a to 4bcd774 Compare December 2, 2024 19:46

Add env cache

31d5198

dfsnow force-pushed the dfsnow/setup-tox-caching branch from 4bcd774 to 31d5198 Compare December 2, 2024 19:50

dfsnow closed this Dec 2, 2024

dfsnow reopened this Dec 2, 2024

Pass UV_CACHE_DIR

9ff3c3f

dfsnow force-pushed the dfsnow/setup-tox-caching branch from bc55777 to 9ff3c3f Compare December 2, 2024 20:44

dfsnow added 7 commits December 2, 2024 15:14

Disable cache pruning

fa7169d

Add full uv workflow and caching

a580397

Finalize GH workflows with uv

c9ce0de

Change uv cache suffixes

cc4dcad

Use system Python for tox

7d2d3a5

Use uv python for tox

10322e7

Drop caching from pages

e311aff

dfsnow changed the title ~~Split tox setup and run~~ Fix tox + uv caching Dec 2, 2024

Disable cache on lint workflow

71ac9f2

dfsnow commented Dec 2, 2024

View reviewed changes

dfsnow had a problem deploying to pypi December 2, 2024 22:15 — with GitHub Actions Failure

dfsnow commented Dec 2, 2024

View reviewed changes

Cache for each Python version

3cda093

dfsnow commented Dec 2, 2024

View reviewed changes

dfsnow marked this pull request as ready for review December 2, 2024 22:28

dfsnow requested a review from wrridgeway as a code owner December 2, 2024 22:28

dfsnow requested a review from jeancochrane December 2, 2024 22:28

jeancochrane approved these changes Dec 2, 2024

View reviewed changes

Drop dedicated linting PR

55e3225

dfsnow force-pushed the dfsnow/setup-tox-caching branch 2 times, most recently from 0d2737c to 55e3225 Compare December 2, 2024 23:34

dfsnow merged commit 494da48 into main Dec 2, 2024
16 checks passed

dfsnow deleted the dfsnow/setup-tox-caching branch December 2, 2024 23:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix tox + uv caching #29

Fix tox + uv caching #29

dfsnow commented Dec 2, 2024 •

edited

Loading

dfsnow Dec 2, 2024

jeancochrane Dec 2, 2024

dfsnow Dec 2, 2024

jeancochrane Dec 2, 2024

dfsnow Dec 2, 2024

dfsnow Dec 2, 2024 •

edited

Loading

jeancochrane Dec 2, 2024

dfsnow Dec 2, 2024

jeancochrane left a comment

jeancochrane Dec 2, 2024

jeancochrane Dec 2, 2024

dfsnow Dec 2, 2024

jeancochrane Dec 2, 2024

jeancochrane Dec 2, 2024

Fix tox + uv caching #29

Fix tox + uv caching #29

Conversation

dfsnow commented Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfsnow Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeancochrane left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dfsnow commented Dec 2, 2024 •

edited

Loading

dfsnow Dec 2, 2024 •

edited

Loading