Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GTFS schedule rewrite: Implement task result base class #79

Open
lauriemerrell opened this issue Sep 1, 2022 · 3 comments
Open

GTFS schedule rewrite: Implement task result base class #79

lauriemerrell opened this issue Sep 1, 2022 · 3 comments
Labels
good first issue Good for newcomers

Comments

@lauriemerrell
Copy link
Contributor

We have a ProcessingOutcome base class, but no JobResult base class. This results in us reimplementing very similar versions of the following code in every since job/task (specifically: the schedule download job, schedule validation, RT parsing, RT validation, schedule unzipping, and schedule conversion to JSON.)

We should add a base class with this and then just subclass it in each instance to avoid copy/pasted code.

class JobResult(PartitionedGCSArtifact):
    bucket: ClassVar[str] 
    table: ClassVar[str]
    partition_names: ClassVar[List[str]]
    outcomes: List[ProcessingOutcome]

    @property
    def successes(self) -> List[ProcessingOutcome]:
        return [outcome for outcome in self.outcomes if outcome.success]

    @property
    def failures(self) -> List[ProcessingOutcome]:
        return [outcome for outcome in self.outcomes if not outcome.success]

    def save(self, fs):
        self.save_content(
            fs=fs,
            content="\n".join(o.json() for o in self.outcomes).encode(),
            exclude={"outcomes"},
        )
@lauriemerrell
Copy link
Contributor Author

We think this may be lower priority than moving on to warehouse work.

@atvaccaro
Copy link
Contributor

Currently we've been using .jsonl.gz for data and .jsonl for outcomes. We could continue this pattern and enforce it in the type; it makes it mildly easier to look at outcomes which really shouldn't be that large anyways.

@lauriemerrell
Copy link
Contributor Author

Maybe revisit this when doing Payments work.

@lauriemerrell lauriemerrell added the good first issue Good for newcomers label Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants