Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added hook to job to delete/recreate during upgrade/rollback #70

Merged
merged 9 commits into from
Jan 10, 2024
Merged

Conversation

jliedy
Copy link
Contributor

@jliedy jliedy commented Oct 9, 2023

Added annotations to the job in order for the migration job to be deleted after a successful run.

Description

In reference to Issue #69 I've added hook annotations to the migration job in order for the job to be deleted and then recreated during a helm upgrade/rollback.

References

For views on how to add hooks to jobs, one can reference the Helm documentation at: https://helm.sh/docs/topics/charts_hooks/

Review Checklist

  • I have clicked on "allow edits by maintainers".
  • I have added documentation for new/changed functionality in this PR or in a PR to openfga.dev [Provide a link to any relevant PRs in the references section above]
  • The correct base branch is being used, if not main
  • I have added tests to validate that the change in functionality is working as expected

@jliedy jliedy changed the title Added hook to job to delete after running Draft - Added hook to job to delete after running Oct 10, 2023
@jliedy
Copy link
Contributor Author

jliedy commented Oct 10, 2023

I've noticed something awkward when deleting a job before the deployment can init is causing the k8s-wait-for initcontainer to not start if you're using v1.7 or later. Gonna see what I can figure out.

@jliedy jliedy changed the title Draft - Added hook to job to delete after running Added hook to job to delete after running Oct 10, 2023
@jliedy
Copy link
Contributor Author

jliedy commented Oct 10, 2023

So with groundnuty/k8s-wait-for v1.6, it didn't cause things to get jammed up with starting the deployment if the job completed and got deleted. v1.7 and later, it DID cause the deployment to fail to init due to the job not being found. Thus, deleting the job at the end of an upgrade/rollback could cause issues with starting the deployment.

Setting the job to be deleted at the beginning of the install/upgrade/rollback process will allow the job to linger as a way to ensure that the deployment initContainer will be able to successfully validate job completion.

groundnuty/k8s-wait-for can remain at v1.6 for flexibility (if the job doesn't exist, the deployment can still init), but using v1.7 and later can help ensure that the new OpenFGA release doesn't actually start until after the db migration job has completed.

Also, groundnuty/k8s-wait-for v1.7 and later uses get and/or list when used in this fashion.

@jliedy jliedy changed the title Added hook to job to delete after running Added hook to job to delete/recreate during upgrade/rollback. Oct 10, 2023
@jliedy jliedy changed the title Added hook to job to delete/recreate during upgrade/rollback. Draft - Added hook to job to delete/recreate during upgrade/rollback. Oct 10, 2023
@jliedy
Copy link
Contributor Author

jliedy commented Oct 10, 2023

Found an issue that by doing this, the job doesn't use the latest image, but the previously deployed one.

@jliedy
Copy link
Contributor Author

jliedy commented Oct 11, 2023

This solution still isn't my favorite. If you deploy with helm using the --wait flag, it will cause the deploy process to fail as the deployment won't start until the migration job runs, and the migration job won't get applied until the deployment succeeds.

@jliedy
Copy link
Contributor Author

jliedy commented Nov 3, 2023

I've gotten this style to work fairly consistently with a separate helm chart that I manage. Able to consistently deploy and update without issue.

@jon-whit
Copy link
Collaborator

jon-whit commented Dec 11, 2023

@jliedy sorry this one has fallen through the cracks. We've been heads down working on some big features in the core OpenFGA server. Would you mind rebasing on main? We can take a final look afterwards. Thanks!

🙇

@jliedy
Copy link
Contributor Author

jliedy commented Dec 12, 2023

I've had my head down in some stuff myself, but we've been running in openshift what I've suggested with the change here. I'll try to get back to this this week with an update.

@jon-whit
Copy link
Collaborator

jon-whit commented Jan 4, 2024

@jliedy just following up? Any updates/progress since our last chat?

@jliedy
Copy link
Contributor Author

jliedy commented Jan 8, 2024

Sorry, no, I stink. Had some personal things that needed to be taken care of.

Lemme go ahead and get cracking on this now.

@jliedy jliedy closed this Jan 8, 2024
jliedy added 6 commits January 8, 2024 16:42
the beginning of the helm upgrade/rollback process.

Update the role rules to allow both list and get to
support using the latest release of
groundnuty/k8s-wait-for

Update groundnuty/k8s-wait-for to use v2.0

Signed-off-by: Jonathan Liedy <[email protected]>
to job config in order for the job to be deleted
when "helm uninstall" is run.

Signed-off-by: Jonathan Liedy <[email protected]>
so it will be able to actually pull updated configmaps/secrets/image references

Signed-off-by: Jonathan Liedy <[email protected]>
k8s-wait-for v2.0.  This does require that the successfully completed
job stay in the kubernetes cluster as the check will fail if the job
does not exist.

Signed-off-by: Jonathan Liedy <[email protected]>
Signed-off-by: Jonathan Liedy <[email protected]>
@jliedy
Copy link
Contributor Author

jliedy commented Jan 8, 2024

This wasn't supposed to be closed.

@jliedy jliedy reopened this Jan 8, 2024
@jliedy jliedy requested a review from a team as a code owner January 8, 2024 22:05
Copy link

linux-foundation-easycla bot commented Jan 8, 2024

CLA Signed

The committers listed above are authorized under a signed CLA.

Copy link

CLA Not Signed

@rhamzeh rhamzeh marked this pull request as draft January 8, 2024 22:27
@rhamzeh
Copy link
Member

rhamzeh commented Jan 8, 2024

Hi @jliedy - thanks!

I converted it to a draft PR. Once you believe it's ready for review, feel free to mark it as such

Signed-off-by: Jonathan Liedy <[email protected]>
@jliedy jliedy marked this pull request as ready for review January 8, 2024 22:42
@jliedy
Copy link
Contributor Author

jliedy commented Jan 8, 2024

I think it's ready to be reviewed. Let me know if you'd prefer me to add the job migration annotations for the helm hooks to the values.yaml file instead of the job.yaml file.

@rhamzeh rhamzeh changed the title Draft - Added hook to job to delete/recreate during upgrade/rollback. feat: added hook to job to delete/recreate during upgrade/rollback Jan 8, 2024
@rhamzeh
Copy link
Member

rhamzeh commented Jan 8, 2024

Thanks! Will refer to the team for review

@jon-whit
Copy link
Collaborator

I think it's ready to be reviewed. Let me know if you'd prefer me to add the job migration annotations for the helm hooks to the values.yaml file instead of the job.yaml file.

@jliedy everything looks good, but I think this would be a good idea as it is more extensible. If you are using the chart and you wish not to have these hooks, then that should be overridable with the values. Mind making that change real quick and then we can merge.

Signed-off-by: Jonathan Liedy <[email protected]>
Signed-off-by: Jonathan Liedy <[email protected]>
@jliedy
Copy link
Contributor Author

jliedy commented Jan 10, 2024

Yeah, I could make an argument for it either way so I figured I'd put the option out there.

Done.

@jon-whit jon-whit merged commit e4377ef into openfga:main Jan 10, 2024
3 checks passed
@jbouchery jbouchery mentioned this pull request Jan 23, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants