[featutre]: Emulate graceful node shutdown logic from core Kubernetes

[fullykubed]

Description

What problem are you trying to solve?

Core Kubernetes supports graceful node shutdown which provides some ordering to node draining during a shutdown operation.

This is helpful when you have system-node-critical pods that provide key capabilities such as log collection or networking to all the other pods on the node. Kubernete's graceful node shutdown logic provides a mechanism to ensure that all normal pods are terminated before the critical pods are terminated.

As Karpenter implements its own draining logic (configurable via the NodePool's terminationGracePeriod field) which differs from how Kubernetes shuts down nodes, this can lead to differences in how pods are terminated when a node is disrupted.

This is important to address for two reasons:

1. Right now it is more difficult than needed to create an orderly shutdown using Karpenter as all pods are terminated at once regardless of their priority class.

2. Since this logic differs from how Kubernetes shuts down nodes, users must now have multiple mental models for how node termination works which adds operational complexity.

I'd recommend the following:

1. Add ordering to the node draining process by evicting non-critical pods before critical pods.
2. As the NodePool's terminationGracePeriod field loosely maps to the kubelet's shutdownGracePeriod already, add another field called terminationGracePeriodCriticalPods that maps to the kubelet's shutdownGracePeriodCriticalPods.

Additionally, I'd recommend that the implementation for this leave the door open to potentially adding the enhanced pod priority graceful node shutdown in the future.

How important is this feature to you?

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If Karpenter contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sftim]

Also see kubernetes/enhancements#4212 (Declarative node maintenance)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten