Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pull class codes from Athena instead of ccao package in land_nbhd_rate ETL #667

Merged
merged 2 commits into from
Dec 6, 2024
Merged

Pull class codes from Athena instead of ccao package in land_nbhd_rate ETL #667

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fc10f4c
Pull class codes from Athena instead of `ccao` package in land_nbhd_r…
jeancochrane Dec 4, 2024
303be38
Convert land_nbhd_rate back to DOS file format
jeancochrane Dec 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions etl/scripts-ccao-data-warehouse-us-east-1/ccao/ccao-land-land_nbhd_rate.R
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
library(arrow)
library(aws.s3)
library(dplyr)
library(noctua)
library(openxlsx)
library(purrr)
library(readr)
Expand All @@ -16,6 +17,8 @@
input_bucket <- file.path(AWS_S3_RAW_BUCKET, "ccao", "land")
output_bucket <- file.path(AWS_S3_WAREHOUSE_BUCKET, "ccao", "land")

AWS_ATHENA_CONN_NOCTUA <- dbConnect(noctua::athena(), rstudio_conn_tab = FALSE)

# Location of remote files
remote_file_raw_nbhd_rate_2022 <- file.path(
input_bucket, "nbhd_rate", "2022.xlsx"
Expand All @@ -26,7 +29,7 @@
remote_file_raw_nbhd_rate_2024 <- file.path(
input_bucket, "nbhd_rate", "2024.xlsx"
)
remote_file_warehouse_nbhd_rate <- file.path(

Check warning on line 32 in etl/scripts-ccao-data-warehouse-us-east-1/ccao/ccao-land-land_nbhd_rate.R

View workflow job for this annotation

GitHub Actions / lint 

file=/github/workspace/etl/scripts-ccao-data-warehouse-us-east-1/ccao/ccao-land-land_nbhd_rate.R,line=32,col=1,[object_length_linter] Variable and function names should not be longer than 30 characters.
output_bucket, "land_nbhd_rate"
)

Expand All @@ -51,8 +54,10 @@
)

# List of regression classes
class <- ccao::class_dict %>%
filter(regression_class) %>%
class <- dbGetQuery(
AWS_ATHENA_CONN_NOCTUA,
"SELECT class_code FROM ccao.class_dict WHERE regression_class"
) %>%
pull(class_code)

# Load the raw workbooks, rename and clean up columns
Expand Down
Loading