You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 3, 2024. It is now read-only.
This data-ingestion pipeline will definitely suffer schema inconsistency errors when the evolution of other components (such as the dialect-map-core models, or the dialect-map-data-diff DiffMessage model) mismatch the adapters and models defined within this repository.
For those cases, it would be great to adapt the current pubsub-job execution mode to add wrongly-formatted messages to some kind of "recoverable folder", where each failed run generates a new "recoverable file". ♻️
Then, create a recovery-job execution mode, which takes as argument the path to the wrongly-formatted messages file that will be parsed and (hopefully) ingested.
The text was updated successfully, but these errors were encountered:
This data-ingestion pipeline will definitely suffer schema inconsistency errors when the evolution of other components (such as the
dialect-map-core
models, or thedialect-map-data-diff
DiffMessage model) mismatch the adapters and models defined within this repository.For those cases, it would be great to adapt the current
pubsub-job
execution mode to add wrongly-formatted messages to some kind of "recoverable folder", where each failed run generates a new "recoverable file". ♻️Then, create a
recovery-job
execution mode, which takes as argument the path to the wrongly-formatted messages file that will be parsed and (hopefully) ingested.The text was updated successfully, but these errors were encountered: