NTD time series data: create warehouse tables #3665
Conversation
|
Warehouse report 📦 Checks/potential follow-upsChecks indicate the following action items may be necessary.
New models 🌱calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_facilities calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_other calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_rolling_stock calitp_warehouse.mart.ntd_funding_and_expenses.fct_capital_expenditures_time_series_total calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_federal calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_local calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_other calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_state calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_capital_total calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_decommissioned_operatingfares calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_decommissioned_operatingother calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_federal calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_local calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_other calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_state calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_operating_total calitp_warehouse.mart.ntd_funding_and_expenses.fct_operating_and_capital_funding_time_series_summary_total calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_drm calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_fares calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_ga calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_nvm calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_total calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_vm calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_opexp_vo calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_pmt calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_upt calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_voms calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_vrh calitp_warehouse.mart.ntd_funding_and_expenses.fct_service_data_and_operating_expenses_time_series_by_mode_vrm calitp_warehouse.mart.gtfs.fct_vehicle_locations_grouped calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__facilities calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__other calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__rolling_stock calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__capital_expenditures_time_series__total calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_federal calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_local calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_other calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_state calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__capital_total calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__decommissioned_operatingfares calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__decommissioned_operatingother calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_federal calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_local calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_other calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_state calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__operating_total calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__operating_and_capital_funding_time_series__summary_total calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__drm calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__fares calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_ga calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_nvm calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_total calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_vm calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__opexp_vo calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__pmt calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__upt calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__voms calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__vrh calitp_warehouse.staging.ntd_funding_and_expenses.stg_ntd__service_data_and_operating_expenses_time_series_by_mode__vrm DAGLegend (in order of precedence)
|
charlie-costanzo
added
the
data-pipeline-ingestion-and-modeling
charlie-costanzo
requested review from
evansiroky,
vevetron and
tiffanychu90
as code owners
charlie-costanzo
force-pushed
the
ntd-time-series-warehouse-tables
branch
from
2e5ff06 to
10c94a6
Compare
| _1992, | ||
| _1993, | ||
| _1994, | ||
| _1995, | ||
| _1996, | ||
| _1997, | ||
| _1998, | ||
| _1999, | ||
| _2000, | ||
| _2001, | ||
| _2002, | ||
| _2003, | ||
| _2004, | ||
| _2005, | ||
| _2006, | ||
| _2007, | ||
| _2008, | ||
| _2009, | ||
| _2010, | ||
| _2011, | ||
| _2012, | ||
| _2013, | ||
| _2014, | ||
| _2015, | ||
| _2016, | ||
| _2017, | ||
| _2018, | ||
| _2019, | ||
| _2020, | ||
| _2021, | ||
| _2022, | ||
| _2023, | ||
| _2023_mode_status, |
There was a problem hiding this comment.
When each year is a column in a database table it is really tedious to make a query that selects every one of these columns to make a metabase query. Is there a way to transform these tables so that we have a single column of year instead of each year being a column? This would create more rows and thus make it easier to make database queries.
There was a problem hiding this comment.
When each year is a column in a database table it is really tedious to make a query that selects every one of these columns to make a metabase query. Is there a way to transform these tables so that we have a single column of
yearinstead of each year being a column? This would create more rows and thus make it easier to make database queries.
Hey @evansiroky – without a doubt, I was definitely planning on adding that to the next round of work on these tables. I was just planning to get these into the warehouse quickly as MVPs in their current form and then do modeling iterations based on what's most pressing/useful immediately, but I can make these modeling changes before merging this PR if preferred
There was a problem hiding this comment.
Charlie, there is an example you can follow to pivot the table and build the way Evan is asking: int_ntd__monthly_ridership_with_adjustments_vrh.sql
| models: | ||
| - name: dim_capital_expenditures_time_series_read_me_data_dictionary | ||
| - name: dim_operating_and_capital_funding_time_series_read_me_data_dictionary | ||
| - name: dim_service_data_and_operating_expenses_time_series_by_mode_read_me_data_dictionary |
There was a problem hiding this comment.
With cost reduction in my mind, I think we don't need these tables with "read_me" on the name, if you query the external table there are a lot of null columns. We would be spending time generating those tables, but they are not useful.
dim_capital_expenditures_time_series_read_me_data_dictionary
dim_operating_and_capital_funding_time_series_read_me_data_dictionary
dim_service_data_and_operating_expenses_time_series_by_mode_read_me_data_dictionary
stg_ntd__capital_expenditures_time_series__read_me_data_dictionary
stg_ntd__operating_and_capital_funding_time_series__read_me_data_dictionary
stg_ntd__service_data_and_operating_expenses_time_series_by_mode__read_me_data_dictionary
| _1992, | ||
| _1993, | ||
| _1994, | ||
| _1995, | ||
| _1996, | ||
| _1997, | ||
| _1998, | ||
| _1999, | ||
| _2000, | ||
| _2001, | ||
| _2002, | ||
| _2003, | ||
| _2004, | ||
| _2005, | ||
| _2006, | ||
| _2007, | ||
| _2008, | ||
| _2009, | ||
| _2010, | ||
| _2011, | ||
| _2012, | ||
| _2013, | ||
| _2014, | ||
| _2015, | ||
| _2016, | ||
| _2017, | ||
| _2018, | ||
| _2019, | ||
| _2020, | ||
| _2021, | ||
| _2022, | ||
| _2023, | ||
| _2023_mode_status, |
There was a problem hiding this comment.
Charlie, there is an example you can follow to pivot the table and build the way Evan is asking: int_ntd__monthly_ridership_with_adjustments_vrh.sql
Description
Following up on #3655, this PR creates new warehouse tables (staging and mart) and associated yml files for the new NTD time series endpoints that we are ingesting.
Type of change
How has this been tested?
Post-merge follow-ups