Broken symlink garbage collection issue in /run/crio/

[andrew-wilson-88]

Describe the bug
Every few weeks we are seeing seemingly broken symlinks to overlay volumes in the /run/crio/ directory. Eventually this leads to the Inodes on the partition being depleted, resulting in new pods not being able to start after being scheduled.

We do have multiple CronJobs that connect to EBS PV's running each minute so it could be an issue in garbage collection as the symlinks seem to volumes which no longer exist.

Has anybody seen similar, or is able provide any further information into how these symlinks tie in to everything, as some seem to remove themselves and others can stay in place for months for pods which haven't been running for some time.

Version
We've seen this behaviour in version 4.8 - 4.11 on AWS installed using the OKD installer.

How reproducible
This has occurred on multiple nodes for over a year.

[aleskandro]

Hello @andrew-wilson-88 , is it possible to get a must-gather from the cluster and the journal of a node that is affected by this problem?

See https://docs.okd.io/latest/support/gathering-cluster-data.html

Also, could you send some partial output of these commands executed in one of the affected nodes?

find /run/crio -type l ! -readable | wc -l # count the broken symlinks
find /run/crio -type l ! -readable # just an excerpt of the folders where  the broken symlinks reside is good

[andrew-wilson-88]

Hi @aleskandro,

Of course, please find the must gather and export of the journal on the links below:
Must-gather:
https://drive.google.com/file/d/1Dnu-uEJbS93wmCRpJmHD0U7BEHW7hRy7/view?usp=sharing

Journal:
https://drive.google.com/file/d/1EoxS8L7foPzM4ODv4XC9RU952mEZYlcA/view?usp=sharing

The output of the commands listed are as follows, I've also added an excerpt of the broken symlinks and where they are targeted for reference:

# find /run/crio -type l ! -readable | wc -l
118724

[root@ip-10-0-175-156 core]# find /run/crio -type l ! -readable | head
/run/crio/d5ecde94fdc7a4a79acd7811ca4f1f33b25faeab34835ec83bf8f1dfa022a498
/run/crio/aa4bcd3b9eadd6c5bf4c492c67fc9ab59b0d88866de3c4f8dc183763793e3ddc
/run/crio/9e2d0293e76bdf8a4e0204bfc7997b8a9f5ff5a4f0a8a015cfe3712b8bb590e0
/run/crio/8d92e03b9d122dc9a9ba54cbfed6e29c42a76ed5e405e1c3f974e86cc4f85d42
/run/crio/35abdf15ec8e92c3575991bd6cd6d6c6815457f24c88b351b105428ce40d37bb
/run/crio/a666380d3e7ab98f59d7aa525444e51872a64c8d69e4b365e148bfbaa116f853
/run/crio/1f0e23bda8e410507c408217a42be4cf1df1ca916b4c58a21474eb4e2c4891dd
/run/crio/7f53937e92c226e61023beb0bfdb6e05813abbecd20b3b5902d740b7ff682a13
/run/crio/a350b67676252e14e5ba4ce65dad3e39e7a48d2e19f57e48a2c48fe6a313dc17
/run/crio/0bfe728cdba0317268f1572067bdb668e0e100a1c2a20ca05224e11f5feb29c7

lrwxrwxrwx.  1 root root  116 Feb 10 14:58 7930f72f4a0dbc23a595e78557286bcc3219a6201240d3d552bcaf3585a46fc4 -> /run/containers/storage/overlay-containers/7930f72f4a0dbc23a595e78557286bcc3219a6201240d3d552bcaf3585a46fc4/userdata
lrwxrwxrwx.  1 root root  116 Feb 10 14:58 a658177dd6203e45e5624116fe0f4f5b1e0bb878600b5514853aee1b5bd7443e -> /run/containers/storage/overlay-containers/a658177dd6203e45e5624116fe0f4f5b1e0bb878600b5514853aee1b5bd7443e/userdata
lrwxrwxrwx.  1 root root  116 Feb 10 14:58 2efb6fcce36f2cff90fa7e09ae1907d50ba1cae2a270b16d0ad718cdea4e9d73 -> /run/containers/storage/overlay-containers/2efb6fcce36f2cff90fa7e09ae1907d50ba1cae2a270b16d0ad718cdea4e9d73/userdata
lrwxrwxrwx.  1 root root  116 Feb 10 14:58 22efe207ea2e901fad570993758b93145f81d490ab336588c55dd050e2c4fd32 -> /run/containers/storage/overlay-containers/22efe207ea2e901fad570993758b93145f81d490ab336588c55dd050e2c4fd32/userdata
lrwxrwxrwx.  1 root root  116 Feb 10 14:58 e74222911e0981984232c70da280f45ee49d91b2e499bb3d7ea1c180e3cd2e2a -> /run/containers/storage/overlay-containers/e74222911e0981984232c70da280f45ee49d91b2e499bb3d7ea1c180e3cd2e2a/userdata

Please let me know if there is anything else I can provide in regards to this matter.

Regards,
Andrew

[japais]

Hello everyone, I'm experiencing the exact same issue on OKD 4.7.0-0.okd-2021-07-03-190901.
As in this case, we have several cronjobs that run quite frequently.

I'm seeing a large number of inodes being used under /run/crio and /run/crio/exits.

[aleskandro]

Hi all, I apologize for the delay in the response. Thanks for the provided info.

This issue seems reproducible in the 4.13 nighlies as well, with the steps in the next.

When pods are deleted, conmon leaks broken symbolic links in /var/run/crio. Those symbolic links are never garbage collected,
leading long-running nodes with a high-rate of pods creation & deletion cycles (e.g., jobs, cronjobs...) to fill up the
available inodes with broken symlinks.

Environment tested with the following versions for crictl, conmon and runc:

sh-4.4# crictl version
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.26.1-6.rhaos4.13.git159cc9c.el8
RuntimeApiVersion:  v1
sh-4.4# conmon --version
conmon version 2.1.6
commit: d8e2824381519d3bc5944944670225c0b66e6e80
sh-4.4# runc --version
runc version 1.1.4
spec: 1.0.2-dev
go: go1.19.4
libseccomp: 2.5.2

Steps to reproduce

0. Ensure the worker is not getting other pods scheduled. For example, taint the node with

NODE=node1
oc adm taint nodes $NODE conmon-bug=value:NoSchedule

1. Create a debug pod to continuously monitor the broken symlinks:

oc debug node/$NODE
chroot /host
watch 'find /run/crio -type l ! -readable | wc -l'

2. Create a pod (use the proper toleration) in a user project:

oc new-project my-project
oc create -f - <<EOF
kind: Pod
apiVersion: v1
metadata:
  generateName: example-
  labels:
    app: hello
spec:
    nodeSelector:
      kubernetes.io/hostname: $NODE
    tolerations:
    - key: "conmon-bug"
      operator: "Exists"
      effect: "NoSchedule"
    containers:
      - name: hello
        securityContext:
          allowPrivilegeEscalation: false
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
          capabilities:
            drop:
              - ALL
        image: image-registry.openshift-image-registry.svc:5000/openshift/cli
        imagePullPolicy: IfNotPresent
        command:
          - /bin/sh
        args:
          - '-c'
          - date; echo Hello from the Kubernetes cluster
    restartPolicy: Never
EOF

3. Delete the created pod with oc delete ...
4. Wait for the garbage collection to be triggered by the kubelet (about 6 mins)
5. Check again the number of broken links as in point 1
6. The old number of broken symlinks should be less than the new one

Instead of waiting at point 3, you can just delete the container with crictl rm in the node that hosts the pod.
I also tried by applying a constant rate of pod creation/deletion and the number of broken symlinks
was always increasing, linearly.

The broken symlink is created by conmon during the creation of the container and that link is not removed when
the container lifecycle ends.

type=CWD msg=audit(1677322413.913:1809): cwd="/run/containers/storage/overlay-containers/dbe22e9dee1edf1919e6c592a69c967c62a9867a583e6cb2523e4a4ec07ee938/userdata"
type=SYSCALL msg=audit(1677322413.913:1809): arch=c000003e syscall=87 success=no exit=-2 a0=5646135afd80 a1=33 a2=5646135afd80 a3=2b items=1 ppid=1 pid=1058162 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="conmon" exe="/usr/bin/conmon" subj=system_u:system_r:container_runtime_t:s0 key=(null)
----
time->Sat Feb 25 10:53:33 2023
type=PROCTITLE msg=audit(1677322413.913:1810): proctitle=2F7573722F62696E2F636F6E6D6F6E002D62002F72756E2F636F6E7461696E6572732F73746F726167652F6F7665726C61792D636F6E7461696E6572732F646265323265396465653165646631393139653663353932613639633936376336326139383637613538336536636232353233653461346563303765653933382F75
type=PATH msg=audit(1677322413.913:1810): item=2 name="/var/run/crio/dbe22e9dee1edf1919e6c592a69c967c62a9867a583e6cb2523e4a4ec07ee938" inode=12028801 dev=00:18 mode=0120777 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:container_var_run_t:s0 nametype=CREATE cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1677322413.913:1810): item=1 name="/var/run/crio/" inode=32000 dev=00:18 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:container_var_run_t:s0 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(1677322413.913:1810): item=0 name="/run/containers/storage/overlay-containers/dbe22e9dee1edf1919e6c592a69c967c62a9867a583e6cb2523e4a4ec07ee938/userdata" nametype=UNKNOWN cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1677322413.913:1810): cwd="/run/containers/storage/overlay-containers/dbe22e9dee1edf1919e6c592a69c967c62a9867a583e6cb2523e4a4ec07ee938/userdata"
type=SYSCALL msg=audit(1677322413.913:1810): arch=c000003e syscall=88 success=yes exit=0 a0=5646135b7490 a1=5646135afd80 a2=5646135afd80 a3=2b items=3 ppid=1 pid=1058162 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="conmon" exe="/usr/bin/conmon" subj=system_u:system_r:container_runtime_t:s0 key=(null)
----
time->Sat Feb 25 10:53:34 2023

[vrutkovs]

Thanks @aleskandro! Since its not OKD specific lets bring it to OCP jira?

[aleskandro]

Hi @vrutkovs here it is: https://issues.redhat.com/browse/OCPBUGS-7962

[aleskandro]

containers/conmon#384 is now merged, we might get the updated version of conmon in the next (4.12) okd release cut.

[andrew-wilson-88]

Thanks so much! That's great

[aleskandro]

Proposed fix: containers/conmon#384

[andrew-wilson-88]

Hello,

I've tried to follow the merge trail but couldn't find if this has been updated in a new release as of yet. Is there a particular version that the patch has been introduced in?

Regards,
Andrew

[aleskandro]

Hi, conmon-2.1.7 is released with the fix since 2weeks and it is available in fedora-coreos rawhide. However, it didn't land in the fedora-coreos RPMs consumed by OKD (fcos:stable) yet.

Hi @vrutkovs anything needed to cherry-pick the fix and make it land in fcos:stable soon?

[vrutkovs]

I'd rather not divert from stable packages too much. iiuc it should land in about 2 weeks there, so lets wait.

If its urgent you can use layering to update this particular package on the cluster OS

[aleskandro]

As an update:

The fix landed for 4.13.0-0.okd-2023-03-29-070729 (fedora-coreos:testing-devel)

bash-5.2# rpm -aq | grep conmon
conmon-2.1.7-2.fc37.x86_64
bash-5.2# 

However, this is not yet available in 4.12 (fedora-coreos:stable):

bash-5.2# rpm -qa | grep conmon
conmon-2.1.6-3.fc37.x86_64
bash-5.2# 

Finally, it needs to land yet on scos (cc @LorbusChris) for the okd-4.14-scos stream:

bash-5.1# rpm -qa | grep conmon
conmon-2.1.6-1.el9.x86_64