Skip to content
This repository has been archived by the owner on Sep 3, 2024. It is now read-only.

Implement 'recovery mode' #17

Open
Sinclert opened this issue Apr 5, 2021 · 0 comments
Open

Implement 'recovery mode' #17

Sinclert opened this issue Apr 5, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@Sinclert
Copy link
Member

Sinclert commented Apr 5, 2021

This data-ingestion pipeline will definitely suffer schema inconsistency errors when the evolution of other components (such as the dialect-map-core models, or the dialect-map-data-diff DiffMessage model) mismatch the adapters and models defined within this repository.

For those cases, it would be great to adapt the current pubsub-job execution mode to add wrongly-formatted messages to some kind of "recoverable folder", where each failed run generates a new "recoverable file". ♻️

Then, create a recovery-job execution mode, which takes as argument the path to the wrongly-formatted messages file that will be parsed and (hopefully) ingested.

@Sinclert Sinclert added the enhancement New feature or request label Apr 5, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant