You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# TODO: detect "created/updated" and adjust values where appropriate
The problem stems from adding columns to a data frame to time-stamp record creation and update. The added columns are by definition not in the data source and they are completely novel on each run of the script. If on run N we get data A and on run N +1 we get data A + data B. We will rewrite all of A with a new timestamp.
I think the fix is to allow the caller to specify a vector of column names that should be ignored in the anti_join, which removes known data. This new parameter might be named columns_to_ignore, ignore_in_update, novel_columns, or some better name than these examples I am throwing up against the wall.
The text was updated successfully, but these errors were encountered:
Ignore specified columns in
update_records
dataframe ofsync_table_2
atredcapcustodian/R/write_data.R
Line 226 in c4293d2
The problem stems from adding columns to a data frame to time-stamp record creation and update. The added columns are by definition not in the data source and they are completely novel on each run of the script. If on run N we get data A and on run N +1 we get data A + data B. We will rewrite all of A with a new timestamp.
I think the fix is to allow the caller to specify a vector of column names that should be ignored in the anti_join, which removes known data. This new parameter might be named
columns_to_ignore
,ignore_in_update
,novel_columns
, or some better name than these examples I am throwing up against the wall.The text was updated successfully, but these errors were encountered: