Persist rejected data table #294
Conversation
| @@ -194,10 +194,10 @@ class VerticaDistributedFilesystemWritePipe(val config: DistributedFilesystemWri | |||
|
|
|||
| def buildCopyStatement(targetTable: String, columnList: String, url: String, rejectsTableName: String, fileFormat: String): String = { | |||
| if (config.mergeKey.isDefined) { | |||
| s"COPY $targetTable FROM '$url' ON ANY NODE $fileFormat REJECTED DATA AS TABLE $rejectsTableName NO COMMIT" | |||
| s"COPY $targetTable FROM '$url' ON ANY NODE $fileFormat REJECTED DATA AS TABLE $rejectsTableName" | |||
There was a problem hiding this comment.
@jeremyp-bq , @jonathanl-bq , do you know what was the original intent of having "NO COMMIT"? Do we have cases when we aggregate several COPY statements in a single transaction?
There was a problem hiding this comment.
Good point, I think this is the only place we are executing a statement but we probably do want to keep a single commit at the end of the transaction. Although the rejects table is never committed when that is the case. However, I can look to create the rejects table in a separate statement.
There was a problem hiding this comment.
I actually don't remember what the original intent was. I probably glanced over this too quickly.
|
Need to reevaluate how to best persist the rejected table data without impacting other functionality, so closing this PR. |
Summary
Persist rejected data table
Description
Previously any rejected rows (rejected data table) was created as a temporary table locally and never committed to the Vertica database. This change means that every time there is at least one rejected row during a write operation it will persist that information in a rejected data table (see the README for the table name).
Note that a new ticket was created (#293) to add configuration options for this functionality later.
Related Issue
#275
Additional Reviewers
@alexr-bq
@alexey-temnikov
@jonathanl-bq
@ravjotbrar