Define GitHub actions workflows for building and testing dbt #50

jeancochrane · 2023-08-03T14:27:35Z

This PR sets up GitHub Actions workflows for two particular cases that we want to support when building and testing dbt:

A fast build-and-test workflow that runs on pull requests and pushes to main branches (build_and_test_dbt)
- This workflow will provide CI/CD when developing against this repo
- Tests will be run against isolated databases that get torn down when the PR is merged
A workflow that runs the full test suite on manual dispatch (test_dbt_models)
- This workflow will provide an interface to allow extraction jobs like service-sqoop-iasworld to run tests after pulling in new data (see e.g.: [Data catalog] Update run.sh to run dbt tests via the GitHub Actions API after import service-sqoop-iasworld#1)
- Tests will run against the production environment

Closes #31.

…scheme

…v vars

…kflow

…test_dbt workflow

…rkflows

…tions-workflows-for-building-the-dbt-dag-and-running-tests

jeancochrane · 2023-08-07T20:14:42Z

This is finally ready for a review! Probably best if both @dfsnow and @wrridgeway take a look, since the workflows introduced in this PR will operate as the runners of our data integrity tests.

dfsnow

Overall amazing work @jeancochrane! This couldn't have been easy to debug. Just a couple outstanding blockers and we're good to go.

.github/actions/install_dbt_requirements/action.yaml

.github/actions/load_environment_variables/action.yaml

.github/scripts/cleanup_dbt_resources.sh

.github/workflows/build_and_test_dbt.yaml

dfsnow · 2023-08-08T03:29:57Z

.github/actions/install_dbt_requirements/action.yaml

+inputs:
+  dbt_project_dir:
+    description: Path to the directory containing the dbt project.
+    required: false
+    default: ./dbt
+  requirements_file_path:
+    description: Path to Python requirements file.
+    required: false
+    default: ./dbt/requirements.txt


I actually didn't know you could re-use in-repo actions. I sillily used a composable workflow in PTAXSIM instead. Will fix: ccao-data/ptaxsim#15. Thanks for teaching me something today!

.github/workflows/build_and_test_dbt.yaml

.github/workflows/test_dbt_models.yaml

dfsnow · 2023-08-08T04:20:03Z

dbt/models/default/schema.yml

@@ -31,15 +31,15 @@ models:
            - pin
            - year
          config:
-            error_if: ">280655"
+            error_if: ">280659"


suggestion (non-blocking): Some of these are likely just under-specified tests, but I agree it's going to be a problem. Grab @wrridgeway this week for a review/debugging session of the current tests. He knows more than anyone but Mirella about the horrors of our changing data.

…e action

This is necessary to support running test_dbt_models against the data-catalog branch, which is our plan while we continue to develop on that long-lived branch.

jeancochrane · 2023-08-08T16:16:45Z

Closing this temporarily to test the new cleanup-dbt-resources workflow!

jeancochrane · 2023-08-08T16:32:58Z

This is ready for another look @dfsnow! I took just about all of your advice; the only thread with an open question is #50 (comment).

jeancochrane · 2023-08-08T16:47:43Z

Actually, hold off on that review @dfsnow -- my testing on the docs branch (#57) is revealing that there may be a problem with the way we're setting things up for the data-catalog branch here.

…et HEAD_REF env var It turns out that we can't override the GITHUB_HEAD_REF variable in a GitHub Actions workflow, which means we can't set it in order to treat the data-catalog branch environment as a CI target. To get around this, this commit changes up the generate_schema_name macro to read from a new HEAD_REF env var instead of GITHUB_HEAD_REF, and then factors out a configure_dbt_environment composite action that our dbt workflows can use to set this environment variable appropriately.

jeancochrane · 2023-08-08T17:57:07Z

Alright @dfsnow, this is ready again! I made one change that wasn't part of our PR discussion after realizing that we can't override the GITHUB_ENV_VAR environment variable in order to run the ci target on the data-catalog branch. More context in the commit message for 60c2bb1:

It turns out that we can't override the GITHUB_HEAD_REF variable in a GitHub Actions workflow,
which means we can't set it in order to treat the data-catalog branch environment as a CI target.

To get around this, this commit changes up the generate_schema_name macro to read from a new
HEAD_REF env var instead of GITHUB_HEAD_REF, and then factors out a configure_dbt_environment
composite action that our dbt workflows can use to set this environment variable appropriately.

dfsnow

Looks great @jeancochrane, thanks for making those changes! Merge at your leisure

jeancochrane added 7 commits August 2, 2023 12:07

Define GitHub workflows for building and testing dbt models

88494fd

Configure AWS credentials in GitHub Actions build and test workflows

997f3e4

Centralize dbt env vars in GitHub Actions workflows

bc0e000

Tweaks to dbt GitHub Actions workflow definition ahead of testing

ff4fd81

Fix linting problems with dbt GitHub actions and workflows

297f7ec

Empty commit to trigger CI build

0396849

Rename local GitHub actions to match expected dir/action.yaml naming …

f833271

…scheme

jeancochrane force-pushed the jeancochrane/31-data-catalog-define-github-actions-workflows-for-building-the-dbt-dag-and-running-tests branch from 4dbf505 to f833271 Compare August 3, 2023 15:37

jeancochrane added 7 commits August 3, 2023 11:08

Add permissions to interact with GitHub OIDC to dbt actions

3fb81f6

Try new format for build and test cache key on CI

5b37bf8

Strip extraneous dollar sign from dbt workflow cache key

e8ec9d0

Try different format for dbt directory paths in workflow env vars

f4ba9de

Rename dbt workflow env vars to avoid collisions with dbt internal en…

8530109

…v vars

Make sure STATE_ARGS env var is never empty in build_and_test_dbt wor…

b26cefc

…kflow

Try new format for reading dbt commands from env vars in GitHub workflow

3ff6d98

jeancochrane force-pushed the jeancochrane/31-data-catalog-define-github-actions-workflows-for-building-the-dbt-dag-and-running-tests branch from 5f43129 to 3ff6d98 Compare August 3, 2023 16:58

Add step to build_and_test_dbt workflow to test dbt installation

0607be7

jeancochrane force-pushed the jeancochrane/31-data-catalog-define-github-actions-workflows-for-building-the-dbt-dag-and-running-tests branch from 7be8890 to 567f63f Compare August 3, 2023 18:03

Try a different quoting scheme for RUN_CMD and TEST_CMD in build_and_…

98a42ef

…test_dbt workflow

jeancochrane force-pushed the jeancochrane/31-data-catalog-define-github-actions-workflows-for-building-the-dbt-dag-and-running-tests branch from 567f63f to 98a42ef Compare August 3, 2023 18:08

jeancochrane added 10 commits August 3, 2023 13:13

Define build/test commands directly instead of via env vars in dbt wo…

03e4dc1

…rkflows

Log all conditional branches in build_and_test_dbt_models workflow

b99d0da

See if removing hyphens from database names appeases dbt-athena on CI

829a456

Merge data-catalog into jeancochrane/31-data-catalog-define-github-ac…

cb0299b

…tions-workflows-for-building-the-dbt-dag-and-running-tests

Temporarily enable dbt debugging to try to figure out AWS permissions

31de4e0

Try reverting dbt schema naming back to kebab_slugify

7b0037a

Remove --debug flag from dbt run call in build_and_test_dbt workflow

9bf6cac

Bump error thresholds for four dbt tests

76db394

Add step to cleanup resources to build_and_test_dbt workflow

4d552aa

Clean up cleanup_dbt_resources.sh script for use in CI

27f556c

jeancochrane requested review from wrridgeway and dfsnow August 7, 2023 20:13

This was referenced Aug 8, 2023

Refactor GHA composable workflow to use Action instead ccao-data/ptaxsim#15

Closed

Switch GHA workflows to use OIDC ccao-data/ptaxsim#16

Closed

dfsnow reviewed Aug 8, 2023

View reviewed changes

Cache dbt and Python requirements in install_dbt_requirements action

cef322a

jeancochrane force-pushed the jeancochrane/31-data-catalog-define-github-actions-workflows-for-building-the-dbt-dag-and-running-tests branch from 4148bb1 to cef322a Compare August 8, 2023 15:13

jeancochrane added 4 commits August 8, 2023 10:30

Use sed to strip comment lines in load_environment_variables composit…

e51c677

…e action

Kebab case dbt build and test workflow names

f9b5d86

Factor out cleanup-dbt-resources into its own workflow

704ed13

Set GITHUB_HEAD_REF var in test_dbt_models workflow

1738374

This is necessary to support running test_dbt_models against the data-catalog branch, which is our plan while we continue to develop on that long-lived branch.

jeancochrane closed this Aug 8, 2023

jeancochrane reopened this Aug 8, 2023

Bump threshold for vw_pin_appeal dbt test failures

672a7d4

jeancochrane requested a review from dfsnow August 8, 2023 16:32

jeancochrane removed request for wrridgeway and dfsnow August 8, 2023 16:46

jeancochrane marked this pull request as draft August 8, 2023 17:11

jeancochrane marked this pull request as ready for review August 8, 2023 17:54

dfsnow self-requested a review August 8, 2023 18:03

dfsnow approved these changes Aug 8, 2023

View reviewed changes

jeancochrane merged commit 9574a38 into data-catalog Aug 8, 2023

jeancochrane deleted the jeancochrane/31-data-catalog-define-github-actions-workflows-for-building-the-dbt-dag-and-running-tests branch August 8, 2023 19:14

jeancochrane mentioned this pull request Aug 8, 2023

[Data catalog] Define GitHub Actions workflows for building the dbt DAG and running tests #31

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Define GitHub actions workflows for building and testing dbt #50

Define GitHub actions workflows for building and testing dbt #50

jeancochrane commented Aug 3, 2023 •

edited

Loading

jeancochrane commented Aug 7, 2023 •

edited

Loading

dfsnow left a comment

dfsnow Aug 8, 2023

dfsnow Aug 8, 2023

jeancochrane commented Aug 8, 2023

jeancochrane commented Aug 8, 2023

jeancochrane commented Aug 8, 2023

jeancochrane commented Aug 8, 2023

dfsnow left a comment

Define GitHub actions workflows for building and testing dbt #50

Define GitHub actions workflows for building and testing dbt #50

Conversation

jeancochrane commented Aug 3, 2023 • edited Loading

jeancochrane commented Aug 7, 2023 • edited Loading

dfsnow left a comment

Choose a reason for hiding this comment

dfsnow Aug 8, 2023

Choose a reason for hiding this comment

dfsnow Aug 8, 2023

Choose a reason for hiding this comment

jeancochrane commented Aug 8, 2023

jeancochrane commented Aug 8, 2023

jeancochrane commented Aug 8, 2023

jeancochrane commented Aug 8, 2023

dfsnow left a comment

Choose a reason for hiding this comment

jeancochrane commented Aug 3, 2023 •

edited

Loading

jeancochrane commented Aug 7, 2023 •

edited

Loading