Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dbt data integrity tests to the model database tables #673

Closed
dfsnow opened this issue Dec 6, 2024 · 1 comment · Fixed by #686
Closed

Add dbt data integrity tests to the model database tables #673

dfsnow opened this issue Dec 6, 2024 · 1 comment · Fixed by #686
Assignees
Labels
bug Something isn't working dbt Related to dbt (tests, docs, schema, etc)

Comments

@dfsnow
Copy link
Member

dfsnow commented Dec 6, 2024

Last week @jeancochrane found a duplicate PIN in one of our model tables (I believe model.assessment_pin). I'm not sure how that snuck in there, but we should take steps to catch such issues in the future.

Let's add dbt data integrity tests to each of the model database sources. We can trigger the tests as part of the weekly cron job and/or via an API call after the completion of a model run (similar to the one called via service-spark-iasworld). We'll mostly want to test for null values and uniqueness.

@dfsnow dfsnow added bug Something isn't working dbt Related to dbt (tests, docs, schema, etc) labels Dec 6, 2024
@Damonamajor
Copy link
Contributor

Damonamajor commented Dec 18, 2024

https://us-east-1.console.aws.amazon.com/s3/buckets/ccao-model-results-us-east-1?region=us-east-1&bucketType=general&prefix=feature_importance/year%3D2024/&showversions=false

feature importance is duplicated for 2024-02-05-sad-tristan.parquet

parameter_search does not seem to have correct keys

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dbt Related to dbt (tests, docs, schema, etc)
Projects
None yet
3 participants