KEP-4563: EvictionRequest API (fka Evacuation)

[atiratree]

• One-line PR description: introduce EvictionRequest API to allow managed graceful pod removal

• Issue link: EvictionRequest API #4563

• Other comments: part of the functionality has been split from KEP-4212: Declarative Node Maintenance #4213 into this KEP

[sftim]

Watch out for the risk of confusing end users.

We already have preemption and eviction and people confuse the two. Or three, because there are two kinds of eviction. And there's disruption in the mix.

Do we want to rename Scheduling, Preemption and Eviction to Scheduling, Preemption, Evacuation and Eviction?

[atiratree]

Good idea, I added a mention which of what kind of eviction I mean here.

Do we want to rename Scheduling, Preemption and Eviction to Scheduling, Preemption, Evacuation and Eviction?

Yes, I think we want to add a new concept there and generally update the docs once we have an alpha.

[alculquicondor]

I'm with Tim here.

Preemption vs Eviction is already quite confusing. And TBH, I couldn't fully understand what the "evacuation" is supposed to solve by reading the summary or motivation.

From Goals:

Changes to the eviction API to support the evacuation process.

If this is already going to be part of the Eviction API, maybe it should be named as a form of eviction. Something like "cooperative eviction" or "eviction with ack" or something along those lines?

[sftim]

I'm all for framing it as another type of eviction; we already have two, so the extra cognitive load for users is not so much a problem.

[atiratree]

@alculquicondor I have updated the summary and goals, I hope it makes more sense now.

I think the name should make the most sense to the person creating the Evacuation (Evacuation Instigator ATM). So CooperativeEviction or EvictionWithAck is a bit misleading IMO. Because from that person's perpective there is no additional step required of them. Only the evacuators and the evacuation controller implement the cooperative evacuation process but this is hidden from the normal user.

My suggestions:

• GracefulEviction (might confuse people if it is associated with graceful pod termination, which it is not)
• SafeEviction (*safer than the API-initiated one for some pods)
• Or just call it Eviction? And tell people to use it instead of the Eviction API endpoint? This might be a bit confusing (at least in the beginning)

[sftim]

We can bikeshed names for the API kind; I'd throw a few of my own into the hat:

• EvictionRequest
• PodEvictionRequest

[atiratree]

I have renamed the API to EvictionRequest to make the term recognizable. A minor disadvantage is that we have to clarify what type of eviction we mean if we say evict (API-initiate eviction, or EvictionRequest)

The rest of the renames are as follows:

Evacuation (noun) -> EvictionRequest / Eviction Process
evacuation (verb) -> request an eviction / terminate / evict / process eviction
Evacuator -> Interceptor
Evacuee -> Pod
Evacuator Class -> Interceptor Class
Evacuation Instigator -> Eviction Requester
Evacuation Controller -> Eviction Request Controller
ActiveEvacuatorClass -> ActiveInterceptorClass
ActiveEvacuatorCompleted -> ActiveInterceptorCompleted
EvacuationProgressTimestamp -> ProgressTimestamp
ExpectedEvacuationFinishTime -> ExpectedInterceptorFinishTime
EvacuationCancellationPolicy -> EvictionRequestCancellationPolicy
FailedEvictionCounter -> FailedAPIEvictionCounter

[sftim]

We have a number of examples of having a SomethingRequest or SomethingClaim API that then causes a something (certificate signing, node provisioning, etc).
Think of TokenRequest (a subresource for a ServiceAccount), or CertificateSigningRequest.

I would confidently and strongly prefer to have an EvictionRequest or PodEvictionRequest API, rather than an Evacuation API kind.

It's easy to teach that we have evictions and than an EvictionRequest is asking for one to happen; it's hard to teach the difference between an eviction and an evacuation.

As a side effect, this makes the feature gate easier to name (eg PodEvictionRequests).

[atiratree]

As you have mentioned, we already have different kinds of eviction. So I think it would be good to use a completely new term to distinguish it from the others.

Also, Evacuation does not always result in eviction (and PDB consultation). It depends on the controller/workload. For some workloads like DaemonSets and static pods, API eviction has never worked before. This could also be very confusing if we name it the same way.

I think Evacuation fits this better because

1. The name is shorter. If we go with EvacuationRequest then the evacuation will become just an abstract term and less recognizable.
2. It seems it will have quite a lot of functionality included (state synchronization between multiple instigators and multiple evacuators, state of the evacuee and evacuation). TokenRequest and CertificateSigningRequest are simpler and not involved in a complex process.

[sftim]

I suggested EvictionRequest so that we don't have to have a section with the (too long) title: Scheduling, Preemption, Evacuation and Eviction. Not EvacuationRequest.

Adding another term doesn't scale so well: it means helping n people understand the difference between evacuation and eviction. It's a scaling challenge where n is not only large, it probably includes quite a few Kubernetes maintainers.

[sftim]

As for CertificateSigningRequest being simple: I don't buy it. There are three controllers, custom signers, an integration with trust bundles, the horrors of ASN.1 and X.509… trust me, it's complicated enough.

[atiratree]

I understand that it will be confusing for people, but that will happen regardless of what term we will use.

My main issue is that evacuation does not directly translate to eviction. Therefore, I think it would be preferable to choose a new term (not necessarily evacuation).

I would like to get additional opinions from people about this. And we will definitely have to come back to this in the API review.

[atiratree]

Should be resolved now: #4565 (comment)

[atiratree]

• I have updated the KEP to include support for multiple evacuators
• Evacuators can now advertise which pods they are able to evacuate, even before the evacuation. The advantage of this approach is that we can trigger an eviction immediately without a delay (was known asacceptDeadlineSeconds) if we do not find an evacuator. I have added a bunch of restrictions to ensure the API cannot be misused.
• Clarified how the Evacuation objects should be named and how the admission should work in general (also for pods). This will ensure a 1:1 mapping between pods and Evacuation.
• Removed the ability to add a full reference of the evacuator because it would be a hassle to synchronize considering the evacuator leader election and multiple evacuatorsin play.

[sftim]

If the descheduler requests an eviction, what thing is being evacuated?

(the node maintenance and cluster autoscaler examples are easier: you're evacuating an entire node)

[atiratree]

A single pod or multiple pods. The descheduler can use it as a new mechanism instead of eviction.

[sftim]

OK, so people typically think of “evacuate” as a near synonym of “drain” - you drain a node, you evacuate a rack or zone full of servers. Saying that you can evacuate a Pod might make people think its containers all get stopped, or just confuse readers. We do need to smooth out how we can teach this.

[atiratree]

It seems it can be used in both scenarios https://www.merriam-webster.com/grammar/can-you-evacuate-people.
Evacuation of containers doesn't make sense because they are tied to the pod lifecycle. But, I guess it could be confusing if we do not make it explicitly clear what we are targeting.

[sftim]

Thing is, Kubernetes users typically - before we accept this KEP - use “evacuate” as a synonym for drain.

I'm (still) fine with the API idea, and still concerned about the naming.

[tuibeovince]

Just to +1 the potential confusion of the term "evacuation".
Is it okay to have a glossary of terms for "evacuation", "eviction", and "drain" (or any other potentially confusing terms) added somewhere in this KEP?

[atiratree]

I can include it in the KEP. And yes, we are going to change the name to something more suitable.

[thockin]

"To evacuate a person" implies "get them out of trouble, to safety" as opposed to "to empty" (as in vacuum). It's not ENTRIELY wrong in this context, but it's not entirely right either.

[atiratree]

I will change the name to EvictionRequest. It was originally chosen to distinguish it from eviction, but there is value in making it familiar to existing concepts.

[atiratree]

The API is renamed to EvictionRequest now, see #4565 (comment) for more details

[atiratree]

KEP Update; notable changes:

• Evacuation has been renamed to EvictionRequest and Evacuator to Interceptor.
• The EvictionRequest is considered complete when the pod terminates (phase Succeeded or Failed). The pod does not need to be deleted if the restartPolicy allows it.
• Improved proposal summary + many small improvements.
• EvictionRequest name should equal to the pod UID.
• Increased constraints for pod interceptor annotations.
• Updated issues.

Added practical use cases/example scenarios:

• Deployment Pod Surge Example
• HorizontalPodAutoscaler Pod Surge Example
• Descheduling and Downscaling Examples
• EvictionRequest Cancellation Examples

[atiratree]

I am out of time for now, but I have to admit I am worried about how this will play out. Coordinating between N arbitrary actors via the API is not somethign we really do anywhere else (AFAIK "classic" DRA was the one place and that is being removed).

Can we pin down some very concrete examples of what REAL evacuators might really do, and how the lifecycle works, especially WRT cancellation of an eviction? You can lose the details and just tell a story. This is significantly complicated and I don't feel able to defend it right now.

@thockin, in the workloads space there are cross-cutting concern issues where such coordination is necessary (PTAL at examples above).

• For example, in point 3 of the Motivation section we see the need to coordinate the ReplicaSet controller (most likely also the Deployment controller), the HPA and the descheduler (or similar component) just on the interceptor side. And this does not take into account other custom interceptors that users may want to implement.
• In the EvictionRequest Cancellation Examples section, we can also see the need for coordination between eviction requesters.

Could this be positioned like "readiness gates". That sort of feels similar.

Such deletion gates could work in a system with well-meaning and diligent actors, but difficult to achieve in practice:

• One of the main goals is not just to help application developers and application-oriented components to implement their use cases. But to also to help cluster administrators to detect misbehaving actors and have a well-defined path to achieve pod termination. So this is why we require a proof of work (status update) to prevent people from using it as just another PDB that blocks cluster upgrades.
• As above, it is difficult to achieve coordination between multiple actors through the use of gates.

[soltysh]

I've read through the initial requirements and assumptions, I haven't read carefully through the Eviction protocol and the contract described in the latter part of this document.

[soltysh]

This case seems identical with the first one. In both cases you're covering a single-replica application. How it got there (always has been, or was scaled down) doesn't matter, imo. So I'd advice to squash the two together.

[paulgmiller]

HPAs are fun here though because they can heal in ways are otherwise left to either deployment/statefulset controllers. So maybe it should be positioned as a consumer of this api?

[atiratree]

Yes, it is similar that it covers scaling of a single replica application, but the impact and followups are different. HPA covers 1-n replicas by default. So I think it is okay to keep it separate.

So maybe it should be positioned as a consumer of this api?

HPA is interesting because it can be both a requester (e.g. during scaling down) and an interceptor (e.g. during node drain). As is shown in the examples in the KEP.

[soltysh]

You can't enforce it in any way, and the API will get used in various, unpredictable ways (see PDB use cases, which you nicely outlined in the sections above). I understand the sentiment, but I don't think it will help here in any way.

[atiratree]

answered in #4565 (comment)

[soltysh]

How are you planning to enforce the progress here? How can you tell that a progress stuck for several minutes is good or bad? Every application will define that differently, so it's hard to define one-size fits all. Even if you require an update every X amount of time, that can be overused by dummy updates, just to block longer/indefinitely.

[atiratree]

We do not need to enforce the progress. This is modelled after leader election. If an interceptor fails to update the EvictionRequest status to report the progress we will choose another interceptor or evict the pod.

Leader election works fine with much lower timeouts. I think we can be more forgiving here, but the 10m mark (allowed minimum) should be sufficient.

that can be overused by dummy updates, just to block longer/indefinitely.

The same could be said about the leader election, which could be used to hijack an existing controller.

The idea behind this is that we should make it very unpleasant to block eviction (proof of work instead of just creating a PDB). And if someone implements this, they might as well implement graceful termination.

Another benefit of this approach is that an administrator can take a look at a very long EvictionRequest and see what InterceptorClass is set on it and can easily track down the misbehaving actor. We can also hook into this with metrics/alerts.

Finally, since this is a normal k8s object, we can use RBAC to restrict and observer access to the EvictionRequest.

[soltysh]

Have you considered improving PDB to provide more insightful status about eviction, which would allow external controllers to watch that status and react accordingly (either by scaling the app up, or doing the actions you're describing above)? This way the whole process could be tracked and synchronized outside of the core resources, and k8s would only expose sufficient information to provide that kind of automation.

[atiratree]

• The main problems are cardinality and synchronization, we have a set of pods guarded by a single PDB and there is a group of actors (requesters and interceptors) interested in the lifecycle of these pods. So tracking the state of the pods in the status would not scale well. An aggregate status would be less useful and would still run into update conflicts.
• High API server load: as this status would only be used on eviction, it will have outdated information. Actors interacting with a PDB will want to have the latest information about the pods (e.g. recalculating descheduling rules according to the cpu/mem utilization) and might do frequent status updates, congesting the API server.
• Not all applications use PDBs. The EvictionRequest API tries to serve as a replacement for eviction and as the main descheduling primitive. This has use cases in cluster and pod autoscaling. For example when used with HPA, we would like to downscale any application/set of pods.

[paulgmiller]

I guess dragging along the pdb to have minAvailable == HPA's current target is a pretty hacky thus the need for a non pdb blocked signal.

[atiratree]

Yes, there are a bunch of problems here, user friendliness, atomicity of actions, and responding to any descheduling (e.g. node drain).

[txomon]

Not sure if it's relevant to the conversation, however I would expect that after this change, the user scenario 2 is solved without the user doing anything, as the maxSurge parameter and the fact that it's a rollout deployment are already specified in the deployment spec.

[atiratree]

Yes, the implementation of the follow-ups is only outlined, here and there should be additional KEPs for each improvement. And the majority of the things proposed should result in an immediate benefit without any user action.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: atiratree
Once this PR has been reviewed and has the lgtm label, please assign soltysh for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-apps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment