Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial dask implementation #10

Closed
kreczko opened this issue Feb 21, 2024 · 1 comment
Closed

Initial dask implementation #10

kreczko opened this issue Feb 21, 2024 · 1 comment
Assignees
Labels
core good first issue Good for newcomers

Comments

@kreczko
Copy link
Contributor

kreczko commented Feb 21, 2024

Adding the ability to execute via Dask, will require modifications to the original workflow:

  • add ability to configure Dask from the config file
  • add extra stage that submits the workflow to dask (this can be done by injecting such a stage into the YAML before it is parsed)
  • add a different executor, parsing the configuration (e.g. n_workers, threads_per_worker)

This example code demonstrates the necessary wrappers to make Dask work with prefect

from prefect import flow, task
from prefect_dask import DaskTaskRunner

....

def dask_flow():
    prefect_future = stage.submit()
    return prefect_future.result()

client = Client()
    wrapper = flow(
        dask_flow,
        task_runner=DaskTaskRunner(
            cluster_kwargs={"n_workers": 2, "threads_per_worker": 1},
        )
    )
    wrapper()
@kreczko
Copy link
Contributor Author

kreczko commented Oct 15, 2024

Fixed in #34

@kreczko kreczko closed this as completed Oct 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant