Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add progress bar to the run function #23

Open
rgriff23 opened this issue Jul 5, 2024 · 0 comments
Open

Add progress bar to the run function #23

rgriff23 opened this issue Jul 5, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@rgriff23
Copy link

rgriff23 commented Jul 5, 2024

The challenge is, there is no straightforward way of telling how many results a flow will produce, given that individual steps or parallel steps can create multiple outputs for any one input. Thus, while it is easy to count how many run results are already complete, we cannot easily translate this to 100%.

For a meaningful progress bar, we need to come up with progress estimates for all steps & propagate these along the flow. I.e. once we know the number of inputs, we estimate the number of outputs of every step to be 1, then update that estimate if the input/output ratio starts to be different. in most cases this will lead to a reasonable progress bar. In some cases, estimates may need to be updated and progress can go from say 80% back to 60%.

We should implement this as an optional argument to the run function, called progress_bar, which defaults to False. This approach ensures that the progress bar is only shown when it is actually useful, avoiding unnecessary clutter when the pipeline runs quickly due to cached results or because the user is testing out a simple pipeline.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants