Skip to content

Commit

Permalink
Merge branch 'main' into feature/paginated-generic-transfer
Browse files Browse the repository at this point in the history
  • Loading branch information
@dabla
dabla authored Jan 1, 2025
2 parents 58c980e + f0003e1 commit fa178af
Show file tree
Hide file tree
Showing 12 changed files with 97 additions and 71 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/finalize-tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -181,15 +181,15 @@ jobs:
- name: "Cleanup repo"
shell: bash
run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm -rf /workspace/*"
- name: Free up disk space
shell: bash
run: ./scripts/tools/free_up_disk_space.sh
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v4
with:
persist-credentials: false
- name: "Cleanup docker"
run: ./scripts/ci/cleanup_docker.sh
- name: Free up disk space
shell: bash
run: ./scripts/tools/free_up_disk_space.sh
- name: "Download all test warning artifacts from the current build"
uses: actions/download-artifact@v4
with:
Expand Down
2 changes: 1 addition & 1 deletion dev/breeze/doc/06_managing_docker_images.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ To load the image from specific job run (for example 12538475388), you can use t

.. code-block:: bash
breeze ci-image load --from-job 12538475388 --python 3.9 --github-token <your_github_token>
breeze ci-image load --from-run 12538475388 --python 3.9 --github-token <your_github_token>
After you load the image, you can reproduce the very exact environment that was used in the CI run by
entering breeze container without mounting your local sources:
Expand Down
114 changes: 70 additions & 44 deletions dev/breeze/doc/ci/07_running_ci_locally.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,11 @@

- [Running the CI Jobs locally](#running-the-ci-jobs-locally)
- [Getting the CI image from failing job](#getting-the-ci-image-from-failing-job)
- [Options and environment variables used](#options-and-environment-variables-used)
- [Basic variables](#basic-variables)
- [Host & GIT variables](#host--git-variables)
- [Test variables](#test-variables)
- [In-container environment initialization](#in-container-environment-initialization)
- [Image build variables](#image-build-variables)
- [Upgrade to newer dependencies](#upgrade-to-newer-dependencies)
- [Host & GIT variables](#host--git-variables)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand Down Expand Up @@ -79,83 +79,109 @@ To load the image from specific PR, you can use the following command:
breeze ci-image load --from-pr 12345 --python 3.9 --github-token <your_github_token>
```

To load the image from specific job run (for example 12538475388),
To load the image from specific run (for example 12538475388),
you can use the following command, find the run id from github action runs.

```bash
breeze ci-image load --from-job 12538475388 --python 3.9 --github-token <your_github_token>
breeze ci-image load --from-run 12538475388 --python 3.9 --github-token <your_github_token>
```

After you load the image, you can reproduce the very exact environment that was used in the CI run by
entering breeze container without mounting your local sources:

```bash
breeze shell --mount-sources skip [OTHER OPTIONS]
breeze shell --mount-sources skip [OPTIONS]
```

And you should be able to run any tests and commands interactively in the very exact environment that
was used in the failing CI run even without checking out sources of the failing PR.
This is a powerful tool to debug and fix CI issues.

You can also build the image locally by checking-out the branch of the PR that was used and running:

```bash
breeze ci-image build
```

You have to be aware that some of the PRs and canary builds use the `--upgrade-to-newer-dependencies` flag
(`UPGRADE_TO_NEWER_DEPENDENCIES` environment variable set to `true`) and they are not using constraints
to build the image so if you want to build it locally, you should pass the `--upgrade-to-newer-dependencies`
flag when you are building the image.

Note however, that if constraints changed for regulare builds and if someone released a new package in PyPI
since the build was run (which is very likely - we have many packages released a day), the image you
build locally might be different than the one in CI, that's why loading image using `breeze ci-image load`
is more reliable way to reproduce the CI build.

If you check-out the branch of the PR that was used, regular ``breeze`` commands will
also reproduce the CI environment without having to rebuild the image - for example when dependencies
changed or when new dependencies were released and used in the CI job - and you will
be able to edit source files locally as usual.
be able to edit source files locally as usual and use your IDE and tools you usually use to develop Airflow.

In order to reproduce the exact job you also need to set the "[OPTIONS]" corresponding to the particular
job you want to reproduce within the run. You can find those in the logs of the CI job. Note that some
of the options can be passed by `--flags` and some via environment variables, for convenience, so you should
take a look at both if you want to be sure to reproduce the exact job configuration. See the next chapter
for summary of the most important environment variables and options used in the CI jobs.

You can read more about it in [Breeze](../README.rst) and [Testing](../../../../contributing-docs/09_testing.rst)

# Options and environment variables used

Depending whether the scripts are run locally via [Breeze](../README.rst) or whether they are run in
`Build Images` or `Tests` workflows they can take different values.
`Build Images` or `Tests` workflows can behave differently.

You can use those variables when you try to reproduce the build locally
(alternatively you can pass those via corresponding command line flags
passed to `breeze shell` command.
You can use those variables when you try to reproduce the build locally - alternatively you can pass
those via corresponding command line flag passed to `breeze shell` command.

## Basic variables

| Variable | Local dev | CI | Comment |
|-----------------------------|-----------|------|------------------------------------------------------------------------------|
| PYTHON_MAJOR_MINOR_VERSION | | | Major/Minor version of Python used. |
| DB_RESET | false | true | Determines whether database should be reset at the container entry. |
| ANSWER | | yes | This variable determines if answer to questions should be automatically set. |
Those variables are controlling basic configuration and behaviour of the breeze command.

## Host & GIT variables
| Variable | Option | Local dev | CI | Comment |
|----------------------------|--------------------------|-----------|------|------------------------------------------------------------------------------|
| PYTHON_MAJOR_MINOR_VERSION | --python | | | Major/Minor version of Python used. |
| BACKEND | --backend | | | Backend used in the tests. |
| INTEGRATION | --integration | | | Integration used in tests. |
| DB_RESET | --db-reset/--no-db-reset | false | true | Determines whether database should be reset at the container entry. |
| ANSWER | --answer | | yes | This variable determines if answer to questions should be automatically set. |

## Test variables

Those variables are used to control the test execution.

| Variable | Option | Local dev | CI | Comment |
|-------------------|---------------------|-----------|----------------------|-------------------------------------------|
| RUN_DB_TESTS_ONLY | --run-db-tests-only | | true in db tests | Whether only db tests should be executed. |
| SKIP_DB_TESTS | --skip-db-tests | | true in non-db tests | Whether db tests should be skipped. |

| Variable | Local dev | CI | Comment |
|-------------------|-----------|------------|-----------------------------------------|
| HOST_USER_ID | Host UID | | User id of the host user. |
| HOST_GROUP_ID | Host GID | | Group id of the host user. |
| HOST_OS | <from os> | linux | OS of the Host (darwin/linux/windows). |
| COMMIT_SHA | | GITHUB_SHA | SHA of the commit of the build is run |

## In-container environment initialization

| Variable | Local dev | CI | Comment |
|---------------------------------|-----------|-----------|-----------------------------------------------------------------------------|
| SKIP_ENVIRONMENT_INITIALIZATION | false (*) | false (*) | Skip initialization of test environment (*) set to true in pre-commits. |
| SKIP_IMAGE_UPGRADE_CHECK | false (*) | false (*) | Skip checking if image should be upgraded (*) set to true in pre-commits. |
| SKIP_PROVIDERS_TESTS | false | false | Skip running provider integration tests. |
| SKIP_SSH_SETUP | false | false (*) | Skip setting up SSH server for tests. (*) set to true in GitHub CodeSpaces. |
| VERBOSE_COMMANDS | false | false | Whether every command executed in docker should be printed. |
Those variables are used to control the initialization of the environment in the container.

| Variable | Option | Local dev | CI | Comment |
|---------------------------------|------------------------------------|-----------|-----------|-----------------------------------------------------------------------------|
| MOUNT_SOURCES | --mount-sources | | skip | Whether to mount the local sources into the container. |
| SKIP_ENVIRONMENT_INITIALIZATION | --skip-enviromnment-initialization | false (*) | false (*) | Skip initialization of test environment (*) set to true in pre-commits. |
| SKIP_IMAGE_UPGRADE_CHECK | --skip-image-upgrade-check | false (*) | false (*) | Skip checking if image should be upgraded (*) set to true in pre-commits. |
| SKIP_PROVIDERS_TESTS | | false | false | Skip running provider integration tests (in non-main branch). |
| SKIP_SSH_SETUP | | false | false (*) | Skip setting up SSH server for tests. (*) set to true in GitHub CodeSpaces. |
| VERBOSE_COMMANDS | | false | false | Whether every command executed in docker should be printed. |

# Image build variables
## Host & GIT variables

| Variable | Local dev | CI | Comment |
|---------------------------------|-----------|-----------|--------------------------------------------------------------------|
| UPGRADE_TO_NEWER_DEPENDENCIES | false | false (*) | Whether dependencies should be upgraded. (*) set in CI when needed |
Those variables are automatically set by Breeze when running the commands locally, but you can override them
if you want to run the commands in a different environment.

# Upgrade to newer dependencies
| Variable | Local dev | CI | Comment |
|---------------|-----------|------------|----------------------------------------|
| HOST_USER_ID | Host UID | | User id of the host user. |
| HOST_GROUP_ID | Host GID | | Group id of the host user. |
| HOST_OS | <from os> | linux | OS of the Host (darwin/linux/windows). |
| COMMIT_SHA | | GITHUB_SHA | SHA of the commit of the build is run |

By default, we are using a tested set of dependency constraints stored in separated "orphan" branches of the airflow repository
("constraints-main, "constraints-2-0") but when this flag is set to anything but false (for example random value),
they are not used and "eager" upgrade strategy is used when installing dependencies. We set it to true in case of direct
pushes (merges) to main and scheduled builds so that the constraints are tested. In those builds, in case we determine
that the tests pass we automatically push latest set of "tested" constraints to the repository. Setting the value to random
value is best way to assure that constraints are upgraded even if there is no change to pyproject.toml
This way our constraints are automatically tested and updated whenever new versions of libraries are released.
(*) true in case of direct pushes and scheduled builds

----

**Thank you** for reading this far. We hope that you have learned a lot about Airflow's CI.
**Thank you** for reading this far. We hope that you have learned a lot about Reproducing Airlfow's CI job locally and CI in general.
Loading

0 comments on commit fa178af

Please sign in to comment.