[Snappi] Handling multi-dut scenario for verify_pause_frame_count in …

…PFC lossless test (sonic-net#13486)

Description of PR
The PFC multi-dut testcases checks the pause frame count transmitted and received by the DUT during the test.

Testcase needed modifications to handle the multiple line card scenario.

Summary:
Fixes # (issue)
Issue sonic-net#12887

Approach
What is the motivation for this PR?
The PFC lossless testcases uses multidut_helper file to start the traffic and verify the results. One of the verifications involves checking the pause frames being sent and received by the DUT.

A single DUT (object duthost) was being sent to the function 'verify_pause_frame_count' to verify the sent and received pause frames. This will work fine for - Single line card single ASIC and Single line card multiple ASIC - because it is single DUT only.

However, this would fail for multiple line card scenario. Here, the PFC receiving and sending line cards are different, and hence the failure.

How did you do it?
In the multidut_helper file, there is now a code to identify 'rx_dut' (Ingress DUT receiving packets from IXIA) and 'tx_dut' (Egress DUT sending packets to IXIA).

Instead of sending single DUT variable to verify_pause_frame_count, it is now passing rx_dut and tx_dut to it. The 'verify_pause_frame_count' will check the incoming pause frame counts on tx_dut and outgoing pause frame counts on rx_dut.

Since line card is same in case of single line card, the 'rx_dut' and 'tx_dut' will be same and hence will continue to work as is.

How did you verify/test it?
Verified with multiple line card scenario test with both 100 and 400Gbps ingress and egress line cards.

Code snapshot:

20:00:16 traffic_generation.verify_pause_frame_co L1509 INFO   | Rx Pause Frames:2154763 on DUT:ixre-egl-board72, port:Ethernet48 and Prio:3
20:00:18 traffic_generation.verify_pause_frame_co L1509 INFO   | Rx Pause Frames:2154763 on DUT:ixre-egl-board72, port:Ethernet48 and Prio:4
20:00:19 traffic_generation.verify_pause_frame_co L1528 INFO   | Tx Pause Frames:1371912 on DUT:ixre-egl-board71, port:Ethernet40 and Prio:3
20:00:20 traffic_generation.verify_pause_frame_co L1528 INFO   | Tx Pause Frames:1371912 on DUT:ixre-egl-board71, port:Ethernet40 and Prio:4

co-authorized by: [email protected]

tests/common/snappi_tests/traffic_generation.py

@@ -578,7 +578,8 @@ def verify_in_flight_buffer_pkts(duthost,
                               format(dropped_packets))
 
 
-def verify_pause_frame_count_dut(duthost,
+def verify_pause_frame_count_dut(rx_dut,
+                                 tx_dut,
                                  test_traffic_pause,
                                  global_pause,
                                  snappi_extra_params):
@@ -587,7 +588,8 @@ def verify_pause_frame_count_dut(duthost,
     on the DUT
 
     Args:
-        duthost (obj): DUT host object
+        rx_dut (obj): Ingress DUT host object receiving packets from IXIA transmitter.
+        tx_dut (obj): Egress DUT host object sending packets to IXIA, hence also receiving PFCs from IXIA.
         test_traffic_pause (bool): whether test traffic is expected to be paused
         global_pause (bool): if pause frame is IEEE 802.3X pause i.e. global pause applied
         snappi_extra_params (SnappiTestParams obj): additional parameters for Snappi traffic
@@ -599,7 +601,7 @@ def verify_pause_frame_count_dut(duthost,
 
     for peer_port, prios in dut_port_config[1].items():  # PFC pause frames received on DUT's egress port
         for prio in prios:
-            pfc_pause_rx_frames = get_pfc_frame_count(duthost, peer_port, prio, is_tx=False)
+            pfc_pause_rx_frames = get_pfc_frame_count(tx_dut, peer_port, prio, is_tx=False)
             # For now, all PFC pause test cases send out PFC pause frames from the TGEN RX port to the DUT TX port,
             # except the case with global pause frames which SONiC does not count currently
             if global_pause:
@@ -616,7 +618,7 @@ def verify_pause_frame_count_dut(duthost,
 
     for peer_port, prios in dut_port_config[0].items():  # PFC pause frames sent by DUT's ingress port to TGEN
         for prio in prios:
-            pfc_pause_tx_frames = get_pfc_frame_count(duthost, peer_port, prio, is_tx=True)
+            pfc_pause_tx_frames = get_pfc_frame_count(rx_dut, peer_port, prio, is_tx=True)
             if test_traffic_pause:
                 pytest_assert(pfc_pause_tx_frames > 0,
                               "PFC pause frames should be transmitted and counted in TX PFC counters for priority {}"

tests/snappi_tests/multidut/pfc/files/multidut_helper.py

@@ -80,6 +80,14 @@ def run_pfc_test(api,
     duthost2 = snappi_extra_params.multi_dut_params.duthost2
     tx_port = snappi_extra_params.multi_dut_params.multi_dut_ports[1]
 
+    # rx_dut is ingress DUT receiving packets.
+    # tx_dut is egress DUT transmitting packets and also receiving PFCs.
+    tx_dut = duthost2
+    rx_dut = duthost1
+    if rx_port['peer_device'] == duthost1.hostname:
+        tx_dut = duthost1
+        rx_dut = duthost2
+
     pytest_assert(testbed_config is not None, 'Fail to get L2/3 testbed config')
 
     stop_pfcwd(duthost1, rx_port['asic_value'])
@@ -263,14 +271,16 @@ def run_pfc_test(api,
                            snappi_extra_params=snappi_extra_params)
 
     # Verify PFC pause frame count on the DUT
-    verify_pause_frame_count_dut(duthost=duthost,
+    verify_pause_frame_count_dut(rx_dut=rx_dut,
+                                 tx_dut=tx_dut,
                                  test_traffic_pause=test_traffic_pause,
                                  global_pause=global_pause,
                                  snappi_extra_params=snappi_extra_params)
 
     # Verify in flight TX lossless packets do not leave the DUT when traffic is expected
     # to be paused, or leave the DUT when the traffic is not expected to be paused
-    verify_egress_queue_frame_count(duthost=duthost,
+    # Verifying the packets on DUT egress, especially for multi line card scenario
+    verify_egress_queue_frame_count(duthost=tx_dut,
                                     switch_flow_stats=switch_flow_stats,
                                     test_traffic_pause=test_traffic_pause,
                                     snappi_extra_params=snappi_extra_params)

tests/snappi_tests/pfc/files/helper.py

@@ -94,6 +94,11 @@ def run_pfc_test(api,
     pytest_assert(port_id is not None,
                   'Fail to get ID for port {}'.format(dut_port))
 
+    # Single linecard and hence rx_dut and tx_dut are the same.
+    # rx_dut and tx_dut are used to verify_pause_frame_count
+    rx_dut = duthost
+    tx_dut = duthost
+
     # Rate percent must be an integer
     bg_flow_rate_percent = int(BG_FLOW_AGGR_RATE_PERCENT / len(bg_prio_list))
     test_flow_rate_percent = int(TEST_FLOW_AGGR_RATE_PERCENT / len(test_prio_list))
@@ -263,7 +268,10 @@ def run_pfc_test(api,
                            snappi_extra_params=snappi_extra_params)
 
     # Verify PFC pause frame count on the DUT
-    verify_pause_frame_count_dut(duthost=duthost,
+    # rx_dut is Ingress DUT receiving traffic.
+    # tx_dut is Egress DUT sending traffic to IXIA and also receiving PFCs.
+    verify_pause_frame_count_dut(rx_dut=rx_dut,
+                                 tx_dut=tx_dut,
                                  test_traffic_pause=test_traffic_pause,
                                  global_pause=global_pause,
                                  snappi_extra_params=snappi_extra_params)