-
Notifications
You must be signed in to change notification settings - Fork 1
BIOS settings of AMD EPYC 7003 series
Always ensure that the selected motherboard, CPU and chipset combination is compatible with the selected COTS NIC card. If you are unsure, please get in touch with readout developers before you test with a potentially incompatible server.
This documentation assumes that you are using an Intel E810-C card, on a Linux RHEL8 based operating system. (Instructions might differ for Mellanox/NVIDIA Connect-X 100G and 10G OCP NICs.)
AMD Server User Manual can be found here
RS FIXME: ADD DMIDECODE -T2 for MOBO PRODUCT ID
- As presented in the server & motherboard user's manual, connect the NIC to a PCIe slot, whith sufficient PCIe version and lanes available.
Ensure that the equipped memory modules are running at their max capable frequency. Either configuration achieves maximum memory bandwidth and lowest latency in SoC P0 Power State mode with 1600 MHz MEMCLK and 1600 MHz FCLK. Might be needed to tune and OC the settings manually. Location of these settings varies based on MOBO/BIOS/Chipset.
Following the DPDK Tuning manual for AMD EPYC ensure the following options:
(Any value not present here but in the manual, are on default factory settings that matches the manual.)
| Name | Value | Description | Location |
|---|---|---|---|
| Local APIC Mode | X2APIC | ||
| SMT Control | Enable | Opposite value than the manual suggest. We want SMT! | |
| AMD:NPS Intel:SNC | TO BE TESTED | Desired Sub-NUMA clustering varies based on the amount of devices connected to the server. | |
| IOMMU | Enabled | For vfio-pci driver and DPDK | |
| Determinism control | Manual | Enabled toggling the next option below | |
| Determinism Slider | Power | Ensure max perf level for each CPU | |
| ACPI SRAT L3 Cache as NUMA domain | Disable | Don't show L3 caches as individual NUMA domains. (Doesn't work due to clashing other option!) | |
| Fixed SOC P-State | P0 | Highest-performing Infinity Fabric P-State | |
| LCLK Setting | 593 MHz for 7002, AUTO for 7003 | Set the RC frequency to be fixed to lower latency between the Northbridge and the rest of the I/O die. | NBIO Common Options -> SMU Common Options -> LCLK Frequency Control |
| L1/L2 Stream HW Prefetcher | Enable | ||
| Core C-States | Enable(C0/C1) | Enables only non-deep, non-powergated sleep states | |
| DF C-States | Disable | Disables sleep states for Data Fabric | |
| xGMI Force Link Width Control | Forced | Enabled option below | |
| xGMI Force Link Width | x2 | Forces link width to the minimum | |
| TSME | Disable | Disable Transparent Secure Memory Encryption which would result with a bit of memory latency penalty. | Memory->UMC->DDR4->Security |
- Setting the NBIO LCLK frequency to fixed 593 MHz caused issues and packet drops between NIC and MOBO/CPU EPYC 7003 series! Explanation of LCLK for reference:
NBIO Link Clock Frequency
The NBIOs (4x per AMD EPYC™ processor) are the serializers/deserializers (also known as "SerDes") that convert and prepare the I/O signals for the processor's 128 external I/O interface lanes (32 per NBIO).
LCLK (short for link clock frequency) controls the link speed of the internal bus that connects the NBIO silicon with the data fabric. All data between the processor and its PCIe lanes flow to the data fabric based on these LCLK frequency settings. The link clock frequency of the NBIO components need to be forced to the maximum frequency for optimal PCIe performance.
For AMD EPYC™ 7002 series processors, this setting cannot be modified via configuration options in the server BIOS alone. Instead, the AMD-IOPM-UTIL (see Section 3.2.3) must be run at every server boot to disable Dynamic Power Management for all PCIe Root Complexes and NBIOs within the system and to lock the logic into the highest performance operational mode.
For AMD EPYC™ 7003 series processors, configuring all NBIOs to be in "Enhanced Preferred I/O" mode is sufficient to enable highest link clock frequency for the NBIO components.