feat(data-pipelines): Allow for starting Dataproc clusters using custom OS images #1577
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rileyhgrant
approved these changes
Aug 20, 2024
| cluster_args_dict = _parse_flags(cluster_args) | ||
| if "image" in cluster_args_dict: | ||
| return start_custom_image_cluster(name, cluster_args_dict) | ||
| # TODO: Add "else" block to confirm with user to use legacy hailctl initialization |
There was a problem hiding this comment.
Just noting a TODO: left here, wasn't sure if you meant to get back to this before merging this PR.
| @@ -58,6 +58,11 @@ The ClinVar pipelines must be run on clusters with an appropriate version of VEP | |||
| ./deployctl dataproc-cluster start <cluster-name> | |||
| ``` | |||
|
|
|||
| NOTE: Starting with `gnomad version 4.1`, it is highly recommended to specify a | |||
| custom OS image that was built for that gnomAD version. To see these images, run | |||
| `gcloud compute images list --filter="family=dataproc-custom-image"`. You can then | |||
There was a problem hiding this comment.
Running this tells me 0 images were found:
❯ gcloud compute images list --filter="family=dataproc-custom-image"
Listed 0 items.
My gcloud config's project is set to exac-gnomad, maybe that's the reason?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds some helper functions within the
./deployctl dataproc-cluster start ...command to facilitate the creation of Dataproc clusters using a custom OS image.The primary advantage of using pre-built images is that they'll provide a standard, consistent environment to start data pipelines with, using the correct/validated configs, dependencies, and versioning so that the user does not have to manage these factors themselves. The Dataproc start cluster time has also been cut down a few minutes.
These helpers are needed because
deployctl's start command useshailctl, which would run redundant commands that have already been executed in the custom OS image.To provide a transition period for our workflows, the
dataproc-cluster start ...command has been updated to make this opt-in; the user must specify the--imageflag with the custom image name they'd like to use. The inclusion of this flag is what triggers running the helper function to start the cluster. If it is not used, as was the case before this update, the originaldeployctlcode will be run for now.This PR also removes
poetryfrom thedata-pipelinedirectory, as we will revert to managing dependencies using therequirements.txtfile located there.Resolves https://github.com/broadinstitute/gnomad-browser-team/issues/64.