IOMMU configuration

Ensure IOMMU is on in the BIOS

AMD: Called IOMMU

Intel: Called VT-d

Kernel parameters for correct IOMMU settings

Needs to be Translated mode:

vi /etc/default/grub

add iommu=nopt

On AMD:

GRUB_CMDLINE_LINUX="<...> amd_iommu=on iommu=nopt"

On Intel:

GRUB_CMDLINE_LINUX="<...> intel_iommu=on"

Then:

grub2-mkconfig -o /boot/grub2/grub.cfg

ATTENTION: With minor kernel 5.14.0-362.13.1.el9_3.x86_64, the kernel arguments passed were not respected by grub2-mkconfig. We needed to use grubby, as explained on RHEL docs.

Example for np04-srv-031:

grubby --update-kernel=ALL --args="nohz_full=2,3,66,67,4,5,68,69,34,35,98,99,36,37,100,101 isolcpus=domain,managed_irq,2,3,66,67,4,5,68,69,34,35,98,99,36,37,100,101"

Check in kernel logs correct IOMMU settings

Look for iommu group additions in the kernel logs:

[root@np02-srv-031 ~]# cat /var/log/messages | grep iommu
...
Oct 10 10:10:25 np04-srv-031 kernel: pci 0000:ff:1e.2: Adding to iommu group 305
Oct 10 10:10:25 np04-srv-031 kernel: pci 0000:ff:1e.3: Adding to iommu group 305
Oct 10 10:10:25 np04-srv-031 kernel: pci 0000:ff:1e.4: Adding to iommu group 305
Oct 10 10:10:25 np04-srv-031 kernel: pci 0000:ff:1e.5: Adding to iommu group 305
Oct 10 10:10:25 np04-srv-031 kernel: pci 0000:ff:1e.6: Adding to iommu group 305
Oct 10 10:10:25 np04-srv-031 kernel: pci 0000:ff:1e.7: Adding to iommu group 305
Oct 10 10:10:25 np04-srv-031 dracut-cmdline[1418]: Using kernel command line parameters:    BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.14.0-362.8.1.el9_3.x86_64 root=/dev/mapper/sysvg-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.md.uuid=df038986:1ad0bc15:29952d5e:ea043c73 rd.lvm.lv=sysvg/root selinux=0 console=tty0 mitigations=off intel_iommu=on iommu=pt nohz_full=2,3,66,67,4,5,68,69,34,35,98,99,36,37,100,101 isolcpus=domain,managed_irq,2,3,66,67,4,5,68,69,34,35,98,99,36,37,100,101

Binding Intel X710 or X722 10GbE NICs with VFIO

We encountered a problem on Alma9 for binding 10Gb Intel NICs with the vfio-pci driver. Found a workaround on StackOverflow.

If VT-D is not enabled in BIOS and kernel command line (grub option), the work around is to use

echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

To be investigated what is this parameter!