Skip to content

Commit

Permalink
Pull class codes from Athena instead of ccao package in `land_nbhd_…
Browse files Browse the repository at this point in the history
…rate` ETL (#667)

* Pull class codes from Athena instead of `ccao` package in land_nbhd_rate script

* Convert land_nbhd_rate back to DOS file format
  • Loading branch information
jeancochrane authored Dec 6, 2024
1 parent 1e65774 commit 8615ccc
Showing 1 changed file with 7 additions and 2 deletions.
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
library(arrow)
library(aws.s3)
library(dplyr)
library(noctua)
library(openxlsx)
library(purrr)
library(readr)
Expand All @@ -16,6 +17,8 @@ AWS_S3_WAREHOUSE_BUCKET <- Sys.getenv("AWS_S3_WAREHOUSE_BUCKET")
input_bucket <- file.path(AWS_S3_RAW_BUCKET, "ccao", "land")
output_bucket <- file.path(AWS_S3_WAREHOUSE_BUCKET, "ccao", "land")

AWS_ATHENA_CONN_NOCTUA <- dbConnect(noctua::athena(), rstudio_conn_tab = FALSE)

# Location of remote files
remote_file_raw_nbhd_rate_2022 <- file.path(
input_bucket, "nbhd_rate", "2022.xlsx"
Expand Down Expand Up @@ -51,8 +54,10 @@ aws.s3::save_object(
)

# List of regression classes
class <- ccao::class_dict %>%
filter(regression_class) %>%
class <- dbGetQuery(
AWS_ATHENA_CONN_NOCTUA,
"SELECT class_code FROM ccao.class_dict WHERE regression_class"
) %>%
pull(class_code)

# Load the raw workbooks, rename and clean up columns
Expand Down

0 comments on commit 8615ccc

Please sign in to comment.