[containerd] Support containerd v2.0.x #11845

mzaian · 2025-01-01T12:49:38Z

What type of PR is this?

/kind container-managers
/kind feature

What this PR does / why we need it:
Add hashes for containerd versions 2.0.[0-1]
containerd 2.0.0 release notes https://github.com/containerd/containerd/releases/tag/v2.0.0
containerd 2.0.1 release notes https://github.com/containerd/containerd/releases/tag/v2.0.1
Make containerd 2.0.1 default
Set containerd_limit_open_file_num to 1048576

Which issue(s) this PR fixes:

Fixes #11836

Does this PR introduce a user-facing change?:

Make containerd 2.0.1 default
Set containerd_limit_open_file_num to 1048576 so it's configurable.

k8s-ci-robot · 2025-01-01T12:49:46Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mzaian

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mzaian]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mzaian · 2025-01-01T12:51:09Z

I did not remove LimitNOFILE from containerd.service.j2 which is removed from the 2.0.x release. There is a discussion here: containerd/containerd#8924 about that.

mzaian · 2025-01-01T12:51:15Z

/ok-to-test

VannTen · 2025-01-01T23:59:34Z

I don't believe we should do this (differ from upstream in this way). I read most of the containerd discussion; and the change seems correct to me. There is the question of systemd pre v240 where LimitNOfile would need an explicit value, but appart from that it seems sane.

yankay · 2025-01-02T01:40:27Z

Hi @mzaian,

I do not recommend upgrading the default version of containerd to v2 for several reasons:

Breaking Changes: Containerd v2 introduces significant breaking changes that require migration. Currently, most Kubernetes users are still on v1.x.
Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.

Given the magnitude of this change, I suggest that Kubespray should consider supporting both containerd v2 and v1 both, while keeping v1.x as the default. Once the upstream project transitions to v2, Kubespray can then update the default to v2.

/hold (for more discuss)

mzaian · 2025-01-02T07:28:56Z

Hi @mzaian,

I do not recommend upgrading the default version of containerd to v2 for several reasons:

Breaking Changes: Containerd v2 introduces significant breaking changes that require migration. Currently, most Kubernetes users are still on v1.x.

Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.

Given the magnitude of this change, I suggest that Kubespray should consider supporting both containerd v2 and v1 both, while keeping v1.x as the default. Once the upstream project transitions to v2, Kubespray can then update the default to v2.

/hold (for more discuss)

Hi @yankay

As we align with the upstream Kubernetes. Let's keep this on-hold for a future Kubespray release then.

VannTen · 2025-01-03T16:47:20Z

- Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details [here](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.32.md?plain=1#L618).

I don't think this is correct, kubernetes support containerd 2.0 since 1.30 [1] I think the containerd releases prefixed with api are about the containerd grpc API [2] not containerd itself. Defaulting to 2.0 2 kubernetes releases after it's supported does not seems that much of a problem IMO [1]: https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support [2]: https://github.com/containerd/containerd/tree/api/v1.7.19/api

yankay · 2025-01-07T05:49:07Z

Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.
I don't think this is correct, kubernetes support containerd 2.0 since 1.30 [1] I think the containerd releases prefixed with api are about the containerd grpc API [2] not containerd itself. Defaulting to 2.0 2 kubernetes releases after it's supported does not seems that much of a problem IMO [1]: https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support [2]: https://github.com/containerd/containerd/tree/api/v1.7.19/api

Thanks @VannTen

/unhold

mzaian · 2025-01-07T06:57:32Z

Upstream Dependency: The upstream Kubernetes project does not have any plans to migrate to containerd v2 at this time, as they are still relying on v1.x. You can check the details here.
I don't think this is correct, kubernetes support containerd 2.0 since 1.30 [1] I think the containerd releases prefixed with api are about the containerd grpc API [2] not containerd itself. Defaulting to 2.0 2 kubernetes releases after it's supported does not seems that much of a problem IMO [1]: https://github.com/containerd/containerd/blob/main/RELEASES.md#kubernetes-support [2]: https://github.com/containerd/containerd/tree/api/v1.7.19/api

Thanks @VannTen

/unhold

Ok good then we can proceed and this will be out with the next release. So we have time to test and fix any issues when they occur.

yankay · 2025-01-07T07:31:16Z

Yes @mzaian 🎉

There some bugs with containerd v2.0 and kubenretes , eg:

containerd 2.0.0 kubernetes bug containerd/containerd#11117

VannTen · 2025-01-07T08:47:02Z

So what about the LimitNOFile change ?

Also, maybe we should use a action required release note and link to containerd 2.0 release ?

mzaian · 2025-01-07T08:49:55Z

So what about the LimitNOFile change ?

Also, maybe we should use a action required release note and link to containerd 2.0 release ?

I will add it again and add a release not for it.

VannTen

Regarding the service file, should we template based on containerd version ?
Aka keep Limit* on 1.7 ?

VannTen · 2025-01-07T14:01:08Z

roles/container-engine/containerd/templates/containerd.service.j2

@@ -30,7 +30,6 @@ RestartSec=5
 # in the kernel. We recommend using cgroups to do container-local accounting.
 LimitNPROC={{ containerd_limit_proc_num }}
 LimitCORE={{ containerd_limit_core }}
-LimitNOFILE={{ containerd_limit_open_file_num }}


I think on systemd v240 we should use 1024:524288 as explained in containerd/containerd#8924 (comment)
(Rhel 8 and derivatives are on 239 if I believe Repology)

Not sure how to check for systemd version though 🤔

How about leaving this out of this PR and I can work separately in investigating this for all other supported operating systems.

Hi @mzaian,

Would it be better to retain LimitNOFILE={{ containerd_limit_open_file_num }} and adjust the default value of containerd_limit_open_file_num to 1048576, similar to what was done in kubernetes/kops#16329?
This way, users can still customize containerd_limit_open_file_num, and avoid to use infinity

Thank you!

VannTen · 2025-01-07T14:03:26Z

roles/container-engine/containerd/templates/containerd.service.j2

@@ -30,7 +30,6 @@ RestartSec=5
 # in the kernel. We recommend using cgroups to do container-local accounting.
 LimitNPROC={{ containerd_limit_proc_num }}
 LimitCORE={{ containerd_limit_core }}
-LimitNOFILE={{ containerd_limit_open_file_num }}
 LimitMEMLOCK={{ containerd_limit_mem_lock }}


This is also not in containerd upstream service file, right ?

We are matching If you check it here: https://github.com/containerd/containerd/blob/main/containerd.service#L30-L33

mzaian · 2025-01-07T17:29:05Z

Regarding the service file, should we template based on containerd version ? Aka keep Limit* on 1.7 ?

I can do this but first I will check if they cherry-picked this change to 1.7.x

k8s-ci-robot requested review from cyclinder and MrFreezeex January 1, 2025 12:49

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 1, 2025

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Jan 1, 2025

mzaian requested review from yankay, VannTen and tico88612 January 1, 2025 12:52

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 2, 2025

pacoxu mentioned this pull request Jan 3, 2025

CI migration Plan for containerd v2.0 kubernetes/test-infra#34063

Open

mzaian force-pushed the support-containerd-2.0.x branch 2 times, most recently from 037b6e9 to fba3798 Compare January 6, 2025 09:56

yankay mentioned this pull request Jan 6, 2025

Migrate the containerd config to deprecate the CRI registry properties #11861

Open

mzaian force-pushed the support-containerd-2.0.x branch from fba3798 to b730cf6 Compare January 6, 2025 13:31

yankay removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 7, 2025

mzaian force-pushed the support-containerd-2.0.x branch from b730cf6 to 5c8dc3e Compare January 7, 2025 13:40

VannTen reviewed Jan 7, 2025

View reviewed changes

[containerd] Support containerd v2.0.x

4954219

mzaian force-pushed the support-containerd-2.0.x branch from 5c8dc3e to 4954219 Compare January 8, 2025 08:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[containerd] Support containerd v2.0.x #11845

[containerd] Support containerd v2.0.x #11845

mzaian commented Jan 1, 2025 •

edited

Loading

k8s-ci-robot commented Jan 1, 2025

mzaian commented Jan 1, 2025

mzaian commented Jan 1, 2025

VannTen commented Jan 1, 2025 via email

yankay commented Jan 2, 2025 •

edited

Loading

mzaian commented Jan 2, 2025

VannTen commented Jan 3, 2025 via email

yankay commented Jan 7, 2025 •

edited

Loading

mzaian commented Jan 7, 2025

yankay commented Jan 7, 2025 •

edited

Loading

VannTen commented Jan 7, 2025

mzaian commented Jan 7, 2025

VannTen left a comment

VannTen Jan 7, 2025

mzaian Jan 7, 2025

yankay Jan 8, 2025 •

edited

Loading

VannTen Jan 7, 2025

mzaian Jan 7, 2025

mzaian commented Jan 7, 2025

[containerd] Support containerd v2.0.x #11845

Are you sure you want to change the base?

[containerd] Support containerd v2.0.x #11845

Conversation

mzaian commented Jan 1, 2025 • edited Loading

k8s-ci-robot commented Jan 1, 2025

mzaian commented Jan 1, 2025

mzaian commented Jan 1, 2025

VannTen commented Jan 1, 2025 via email

yankay commented Jan 2, 2025 • edited Loading

mzaian commented Jan 2, 2025

VannTen commented Jan 3, 2025 via email

yankay commented Jan 7, 2025 • edited Loading

mzaian commented Jan 7, 2025

yankay commented Jan 7, 2025 • edited Loading

VannTen commented Jan 7, 2025

mzaian commented Jan 7, 2025

VannTen left a comment

Choose a reason for hiding this comment

VannTen Jan 7, 2025

Choose a reason for hiding this comment

mzaian Jan 7, 2025

Choose a reason for hiding this comment

yankay Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

VannTen Jan 7, 2025

Choose a reason for hiding this comment

mzaian Jan 7, 2025

Choose a reason for hiding this comment

mzaian commented Jan 7, 2025

mzaian commented Jan 1, 2025 •

edited

Loading

yankay commented Jan 2, 2025 •

edited

Loading

yankay commented Jan 7, 2025 •

edited

Loading

yankay commented Jan 7, 2025 •

edited

Loading

yankay Jan 8, 2025 •

edited

Loading