Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAINT - CI improvements (security and maintenance) #2077

Open
wants to merge 33 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
890ee6f
:construction_worker: Update release workflow
trallard Nov 21, 2024
e89a490
:construction_worker: Add check for docs links
trallard Nov 21, 2024
865158a
:wrench: Add linkcheck for docs
trallard Nov 21, 2024
e93cf6c
:construction_worker: Separate docs CI into another workflow
trallard Nov 21, 2024
0095423
:wrench: Add ignores and redirects to conf
trallard Nov 21, 2024
e70a1d3
:pencil2: Fix URLs in the docs
trallard Nov 22, 2024
18e2198
:wrench: Update tox command for linkchecks
trallard Nov 22, 2024
53dafed
Remove needs - CI workflow
trallard Nov 25, 2024
74f4a1a
Merge branch 'main' into trallard/maintenance-ql-improvements
trallard Dec 10, 2024
76b95ca
:closed_lock_with_key: Change workflow trigger to workflow_call
trallard Dec 10, 2024
6e7851c
:lock: Ensure proper variable escaping in docs workflow
trallard Dec 10, 2024
3e0899d
:arrow_up: Upgrade runners OS and Python versions
trallard Dec 10, 2024
d61186f
Add coverage required permissions
trallard Dec 10, 2024
e9bd826
Add actions scope to permissions:
trallard Dec 10, 2024
21d7a24
:construction_worker: Add GitHub attestations to release workflow
trallard Dec 10, 2024
e560340
:lock: Add static analysis for GH actions with Zizmor
trallard Dec 10, 2024
9b31a14
:construction_worker: Pin our reusable action to a hash
trallard Dec 11, 2024
560ecfe
Add environment and pinned actions to release workflow
trallard Dec 11, 2024
2a7fca4
Add more comments to workflows
trallard Dec 12, 2024
8ecc548
:fire: Remove accidentally commited file
trallard Dec 12, 2024
3864d38
Remove setup-dev action
trallard Dec 12, 2024
741131c
Fix more URLs
trallard Dec 12, 2024
3bb97dd
Remove pin in STB
trallard Dec 12, 2024
1121c37
Change dependabot frequency
trallard Dec 16, 2024
a7e2f69
Finish changing to SHA for GitHub actions
trallard Dec 16, 2024
90f9d1a
Update linkcheck config
trallard Dec 16, 2024
9adae6d
Ignore unsplash for link check
trallard Dec 16, 2024
afaa316
Merge branch 'main' into trallard/maintenance-ql-improvements
trallard Jan 14, 2025
ccb12a1
Ignore known warnings in linkcheck
trallard Jan 15, 2025
028f584
Update actions
trallard Jan 15, 2025
a6e055c
Separate a11y-tests in CI to expand the testing matrix
trallard Jan 15, 2025
48d47e3
Fix tox call for a11y tests
trallard Jan 15, 2025
5685060
Remove unnecessary posargs from tox commands
trallard Jan 15, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/actions/set-dev-env/action.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
# Reusable action to set our PST development environment
# DO NOT use for release jobs since we cache dependencies
name: Setup PST CI environment
description: Create a PST dev environment

Expand Down
4 changes: 2 additions & 2 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
interval: "monthly"
labels:
- "tag: dependencies"
- "tag: CI"
# npm
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
interval: "monthly"
labels:
- "tag: dependencies"
- "tag: javascript"
Expand Down
152 changes: 49 additions & 103 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
# Testing (pytest, a11y-tests), profiling, and coverage checks for PST
name: continuous-integration

# Concurrency group that uses the workflow name and PR number if available
Expand All @@ -21,6 +22,9 @@ on:
branches:
- main
pull_request:
branches:
- "*"
# allows this to be used as a composite action in other workflows
workflow_call:
# allow manual triggering of the workflow, while debugging
workflow_dispatch:
Expand All @@ -31,10 +35,12 @@ jobs:
strategy:
fail-fast: true
matrix:
# https://github.com/actions/runner-images
# macos-14==latest
# ubuntu-20.04==latest
os: ["ubuntu-latest", "ubuntu-24.04", "macos-14", "windows-latest"]
python-version: ["3.9", "3.10", "3.11", "3.12"]
# ubuntu-24.04==latest
# windows-2022==latest
os: ["ubuntu-latest", "ubuntu-22.04", "macos-14", "windows-latest"]
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
sphinx-version: [""]
include:
# oldest Python version with the oldest Sphinx version
Expand All @@ -58,12 +64,16 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false

- name: "Setup CI environment πŸ› "
uses: ./.github/actions/set-dev-env
uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
with:
python-version: ${{ matrix.python-version }}
pandoc: true

- name: "Run tests βœ…"
shell: bash
run: |
Expand All @@ -77,119 +87,40 @@ jobs:
else
python -Im tox run -e compile-assets,i18n-compile,py$(echo ${{ matrix.python-version }} | tr -d .)-tests
fi

- name: "Upload coverage data to GH artifacts πŸ“€"
if: matrix.python-version == '3.12' && matrix.os == 'ubuntu-latest' && matrix.sphinx-version == 'dev'
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
with:
name: coverage-data-${{ matrix.python-version }}
path: .coverage
if-no-files-found: ignore
include-hidden-files: true

# Only run accessibility tests on the latest Python version (3.12) and Ubuntu
a11y-tests:
name: "a11y-tests (ubuntu-latest, 3.12)"
runs-on: ubuntu-latest
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@v4
- name: "Setup CI environment πŸ› "
uses: ./.github/actions/set-dev-env
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
pandoc: true
graphviz: true
- name: "Run accessibility tests with playwright 🎭"
# build PST, build docs, then run a11y-tests
run: python -Im tox run -m a11y

# Build our docs (PST) on major OSes and check for warnings
build-site:
name: "build PST docs"
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.12"]
include:
# oldest Python version with the oldest Sphinx version
- os: ubuntu-latest
python-version: "3.9"
sphinx-version: "6.1"
runs-on: ${{ matrix.os }}
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@v4
- name: "Setup CI environment πŸ› "
uses: ./.github/actions/set-dev-env
with:
python-version: ${{ matrix.python-version }}
pandoc: true
graphviz: true
- name: "Build docs and check for warnings πŸ“–"
shell: bash
run: |
# check if there is a specific Sphinx version to build with
# example substitution: tox run -e py39-sphinx61-docs
if [ -n "${{matrix.sphinx-version}}" ]; then
python -Im tox run -e py$(echo ${{ matrix.python-version }} | tr -d .)-sphinx$(echo ${{ matrix.sphinx-version }} | tr -d .)-docs
# build with the default Sphinx version
# example substitution: tox run -e py312-docs
else
python -Im tox run -e py$(echo ${{ matrix.python-version }} | tr -d .)-docs
fi

# Run Lighthouse audits on the built site (kitchen-sink only)
lighthouse-audit:
needs: build-site
runs-on: ubuntu-latest
env:
DOCS_DIR: "audit"
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@v4
- name: "Setup CI environment πŸ› "
uses: ./.github/actions/set-dev-env
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: "Copy kitchen sink to a tiny site"
run: |
mkdir -p ${{ env.DOCS_DIR }}/site
cp -r docs/examples/kitchen-sink ${{ env.DOCS_DIR }}/site/kitchen-sink
printf "Test\n====\n\n.. toctree::\n\n kitchen-sink/index\n" > ${{ env.DOCS_DIR }}/site/index.rst
echo 'html_theme = "pydata_sphinx_theme"' > ${{ env.DOCS_DIR }}/site/conf.py
echo '.. toctree::\n :glob:\n\n *' >> ${{ env.DOCS_DIR }}/site/index.rst

# build docs without checking for warnings
python -Im tox run -e docs-no-checks

- name: "Audit with Lighthouse πŸ”¦"
uses: treosh/lighthouse-ci-action@v12
with:
configPath: ".github/workflows/lighthouserc.json"
temporaryPublicStorage: true
uploadArtifacts: true
runs: 3 # Multiple runs to reduce variance

coverage:
name: "Check coverage"
needs: run-pytest
runs-on: ubuntu-latest
# avoid running this on schedule, releases, or workflow_call
if: github.event_name != 'schedule' && github.event_name != 'release' && github.event_name != 'workflow_call'
# avoid running this on schedule, releases, workflow_call, or workflow_dispatch
if: github.event_name != 'schedule' && github.event_name != 'release' && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch'
permissions:
contents: write
pull-requests: write
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false

- name: "Setup CI environment πŸ› "
uses: ./.github/actions/set-dev-env
uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}

- run: python -Im pip install --upgrade coverage[toml]

- name: "Download coverage data πŸ“₯"
uses: actions/download-artifact@v4
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16
with:
pattern: coverage-data-*
merge-multiple: true
Expand All @@ -207,45 +138,60 @@ jobs:
python -Im coverage report --fail-under=80

- name: "Upload HTML report if check failed πŸ“€"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
with:
name: html-report
path: htmlcov
if: ${{ failure() }}

# seems we need to call this from the main CI workflow first
- name: "Coverage comment πŸ’¬"
uses: py-cov-action/python-coverage-comment-action@v3
uses: py-cov-action/python-coverage-comment-action@b2eb38dd175bf053189b35f738f9207278b00925
id: coverage_comment
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: "Store Pull Request comment to be posted πŸ“€"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
if: steps.coverage_comment.outputs.COMMENT_FILE_WRITTEN == 'true'
with:
# leave default names
name: python-coverage-comment-action
path: python-coverage-comment-action.txt

profiling:
needs: [build-site, run-pytest]
needs: [run-pytest]
runs-on: ubuntu-latest
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@v4
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false

- name: "Setup CI environment πŸ› "
uses: ./.github/actions/set-dev-env
uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
with:
# 3.12 is not supported by py-spy yet
python-version: "3.11"

- name: "Run profiling with py-spy πŸ•΅οΈβ€β™‚οΈ"
# profiling needs to be run as sudo
run: python -m tox run -e py311-profile-docs -- -o docbuild_profile.svg
continue-on-error: true

- name: "Upload profiling data to GH artifacts πŸ“€"
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08
with:
name: profile-results
path: docbuild_profile.svg
if-no-files-found: ignore

# Calling the coverage-comment action from the main CI workflow
# we might want to pin the SHA once merged
coverage-comment:
uses: ./.github/workflows/coverage.yml
needs: [coverage]
permissions:
contents: write
pull-requests: write
actions: read
54 changes: 54 additions & 0 deletions .github/workflows/a11y.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: a11y-tests

# Concurrency group that uses the workflow name and PR number if available
# or commit SHA as a fallback. If a new build is triggered under that
# concurrency group while a previous build is running it will be canceled.
# Repeated pushes to a PR will cancel all previous builds, while multiple
# merges to main will not cancel.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true

env:
FORCE_COLOR: "1" # Make tools pretty
DEFAULT_PYTHON_VERSION: "3.12" # keep in sync with tox.ini
PIP_DISABLE_PIP_VERSION_CHECK: "1" # Don't check for pip updates

permissions: {}

on:
push:
branches:
- main
pull_request:
branches:
- "*"
# allows this to be used as a composite action in other workflows
workflow_call:
# allow manual triggering of the workflow, while debugging
workflow_dispatch:

jobs:
a11y-tests:
strategy:
fail-fast: true
matrix:
os: ["ubuntu-latest", "ubuntu-22.04", "macos-14", "windows-latest"]
browser: ["firefox", "chromium"]
runs-on: ${{ matrix.os }}
steps:
- name: "Checkout repository πŸ›Ž"
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
with:
persist-credentials: false

- name: "Setup CI environment πŸ› "
uses: pydata/pydata-sphinx-theme/.github/actions/set-dev-env@01731d0cc57768b9eff1c97f38909932ecd7e7d1
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
pandoc: true
graphviz: true

- name: "Run accessibility tests with playwright 🎭"
# build PST, build docs, then run a11y-tests
run: python -Im tox run -e compile-assets,i18n-compile,py312-docs,a11y-tests -- --browser ${{ matrix.browser }}
17 changes: 8 additions & 9 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
@@ -1,14 +1,11 @@
name: Post coverage comment

on:
workflow_run:
workflows: ["continuous-integration"]
types:
- completed
workflow_call:

jobs:
test:
name: "Run tests & display coverage"
name: "Display code coverage"
runs-on: ubuntu-latest
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
permissions:
Expand All @@ -18,15 +15,17 @@ jobs:
steps:
- name: "Get the triggering workflow run details"
id: get-run
uses: octokit/request-action@v2.x
uses: octokit/request-action@dad4362715b7fb2ddedf9772c8670824af564f0d
with:
route: GET /repos/${{ github.repository }}/actions/runs/${{ github.event.workflow_run.id }}
route: GET /repos/"${REPO}"/actions/runs/"${WORKFLOW_RUN_ID}"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPO: ${{ github.repository }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}

# this needs the .coverage file so we download from the CI workflow artifacts
- name: "Download coverage data πŸ“₯"
uses: actions/download-artifact@v4
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16
with:
pattern: coverage-data-*
merge-multiple: true
Expand All @@ -37,7 +36,7 @@ jobs:
run: ls -R

- name: "Post coverage comment"
uses: py-cov-action/python-coverage-comment-action@v3
uses: py-cov-action/python-coverage-comment-action@b2eb38dd175bf053189b35f738f9207278b00925
with:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_PR_RUN_ID: ${{ github.event.workflow_run.id }}
Loading
Loading