Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Swap to Hamilton DAG #34

Merged
merged 26 commits into from
Oct 15, 2024
Merged

Swap to Hamilton DAG #34

merged 26 commits into from
Oct 15, 2024

Conversation

kreczko
Copy link
Contributor

@kreczko kreczko commented Oct 15, 2024

After careful consideration, switching from prefect to Hamilton due to

  • simpler API for workflow conversion
  • easy caching and Dask configuration
  • better virtualisation and the ability to import flows

This PR:

  • removes prefect
  • renamed stages to tasks
  • adds Workflow disk persistence
  • switches to Python > 3.11 (hashlib constraint)

TODO: update docs for Hamilton usage

kreczko added 25 commits March 20, 2024 14:37
@kreczko
docs: add placeholders for workflow conversions
0a1a9d0
@kreczko
docs: add orchestration and rephrase existing docs
bfc69da
@kreczko
docs: add Snakemake reference
03e189e
@kreczko
pyproject: replace prefect with hamilton and explicit dask dependency
a371bd0
@kreczko
rename stage -> task
b34cbfa
@kreczko
pyproject: add pydantic as new dependency
325195d
@kreczko
feat(cli): reduce verbosity of lint subcommand
da6a653
@kreczko
docs: change stages -> tasks
ef24d45
@kreczko
feat: add metadata to FlowConfig
89421f8
@kreczko
fix(LocalBashOperator): repr should put bash_command in quotes for pr…
dfaeecf 
…oper inspection of source
@kreczko
CI: add python 3.12 for testing
f435991
@kreczko
FlowConfig: add optional metadata
eba062a
@kreczko
build: add python 3.13 to list
a4b436e
@kreczko
build: add dill dependency
b3f1fa2
@kreczko
build: drop python 3.10 due to missing hashlib.file_digest
24508da
@kreczko
feat(workflow): add disk persistence
51a2ffc
@kreczko
tests: stages -> tasks
930d2a4
@kreczko
tests: BashOperator -> LocalBashOperator
1b9038f
@kreczko
test: add workflow.task_names
698a8fa
@kreczko
feat(workflow): add disk persistance
27e4b45
@kreczko
refactor: replace prefect with hamilton
bb85246
@kreczko
fix(cli): add version_base and metadata for config, add hamilton DAG
0afb1e9
@kreczko
feat: align Hamilton cache path with workflow save path
5d35ca2
@kreczko
feat(orch): add adapters for dask and local
b8a9931
@kreczko
fix: pylint errors
715fd3d
@kreczko kreczko force-pushed the kreczko-issue-10-part-2 branch from 581e96f to 715fd3d Compare October 15, 2024 14:57
@kreczko kreczko merged commit 065f4a9 into main Oct 15, 2024
9 checks passed
@kreczko kreczko deleted the kreczko-issue-10-part-2 branch October 15, 2024 15:05
@kreczko kreczko mentioned this pull request Oct 15, 2024
Closed
@kreczko kreczko mentioned this pull request Oct 15, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kreczko