ARP resolution fails with device emulation on 1G ports

[pstavirs]

Topology: G5(lan1) ----- G8(lan4)

For testing purposes these are back-to-back connected, there is expected to be a DUT in between them.

G5 sends ARP requests, G8 replies with ARP replies but G5 does not receive them for some reason

srivatsp@EXA8:~$ sudo tshark -i lan1
Running as user "root" and group "root". This could be dangerous.
Capturing on 'lan1'
    1 0.000000000 3c:00:05:26:7a:3a ? Broadcast    ARP 42 Who has 100.1.1.1? Tell 100.1.1.100
    2 0.000063070 3c:00:05:26:7a:3e ? Broadcast    ARP 42 Who has 100.1.1.1? Tell 100.1.1.104
    3 0.000095200 3c:00:05:26:7a:3b ? Broadcast    ARP 42 Who has 100.1.1.1? Tell 100.1.1.101
    4 0.000126620 3c:00:05:26:7a:3c ? Broadcast    ARP 42 Who has 100.1.1.1? Tell 100.1.1.102
    5 0.000157760 3c:00:05:26:7a:3d ? Broadcast    ARP 42 Who has 100.1.1.1? Tell 100.1.1.103
    6 0.005966220 3c:00:05:26:7a:3a ? Broadcast    ARP 42 Who has 100.1.1.200? Tell 100.1.1.100
    7 0.011663230 3c:00:05:26:7a:3b ? Broadcast    ARP 42 Who has 100.1.1.201? Tell 100.1.1.101
    8 0.017320770 3c:00:05:26:7a:3c ? Broadcast    ARP 42 Who has 100.1.1.202? Tell 100.1.1.102
    9 0.022936050 3c:00:05:26:7a:3d ? Broadcast    ARP 42 Who has 100.1.1.203? Tell 100.1.1.103
   10 0.028575160 3c:00:05:26:7a:3e ? Broadcast    ARP 42 Who has 100.1.1.204? Tell 100.1.1.104

srivatsp@EXA8:~$ sudo tshark -i lan4
[sudo] password for srivatsp:
Running as user "root" and group "root". This could be dangerous.
Capturing on 'lan4'
   17 123.177125930 3c:00:05:26:7a:3a ? Broadcast    ARP 60 Who has 100.1.1.1? Tell 100.1.1.100
   18 123.177146000 3c:00:05:26:7a:3e ? Broadcast    ARP 60 Who has 100.1.1.1? Tell 100.1.1.104
   19 123.177180390 3c:00:05:26:7a:3b ? Broadcast    ARP 60 Who has 100.1.1.1? Tell 100.1.1.101
   20 123.177211210 3c:00:05:26:7a:3c ? Broadcast    ARP 60 Who has 100.1.1.1? Tell 100.1.1.102
   21 123.177242330 3c:00:05:26:7a:3d ? Broadcast    ARP 60 Who has 100.1.1.1? Tell 100.1.1.103
   22 123.183077590 3c:00:05:26:7a:3a ? Broadcast    ARP 60 Who has 100.1.1.200? Tell 100.1.1.100
   23 123.188774390 3c:00:05:26:7a:3b ? Broadcast    ARP 60 Who has 100.1.1.201? Tell 100.1.1.101
   24 123.194431960 3c:00:05:26:7a:3c ? Broadcast    ARP 60 Who has 100.1.1.202? Tell 100.1.1.102
   25 123.200046990 3c:00:05:26:7a:3d ? Broadcast    ARP 60 Who has 100.1.1.203? Tell 100.1.1.103
   26 123.205326810 74:00:08:1a:4d:53 ? 3c:00:05:26:7a:3a ARP 42 100.1.1.200 is at 74:00:08:1a:4d:53
   27 123.205445060 74:00:08:1a:4d:54 ? 3c:00:05:26:7a:3b ARP 42 100.1.1.201 is at 74:00:08:1a:4d:54
   28 123.205559060 74:00:08:1a:4d:55 ? 3c:00:05:26:7a:3c ARP 42 100.1.1.202 is at 74:00:08:1a:4d:55
   29 123.205672340 74:00:08:1a:4d:56 ? 3c:00:05:26:7a:3d ARP 42 100.1.1.203 is at 74:00:08:1a:4d:56
   30 123.205689330 3c:00:05:26:7a:3e ? Broadcast    ARP 60 Who has 100.1.1.204? Tell 100.1.1.104
   31 123.308668490 74:00:08:1a:4d:57 ? 3c:00:05:26:7a:3e ARP 42 100.1.1.204 is at 74:00:08:1a:4d:57

The ARP replies above sent by lan4 are not received by lan1. Here's packet capture dump on the CPU port enp5s0f1 -

srivatsp@EXA8:~$ sudo tshark -i enp5s0f1
[sudo] password for srivatsp:
Running as user "root" and group "root". This could be dangerous.
Capturing on 'enp5s0f1'
   30 123.177126520 3c:00:05:26:7a:3a ? Broadcast    0x4008 46 Ethernet II
   31 123.177169780 3c:00:05:26:7a:3a ? Broadcast    0xc020 64 Ethernet II
   32 123.177181810 3c:00:05:26:7a:3e ? Broadcast    0x4008 46 Ethernet II
   33 123.177189850 3c:00:05:26:7a:3e ? Broadcast    0xc020 64 Ethernet II
   34 123.177213560 3c:00:05:26:7a:3b ? Broadcast    0x4008 46 Ethernet II
   35 123.177224240 3c:00:05:26:7a:3b ? Broadcast    0xc020 64 Ethernet II
   36 123.177244930 3c:00:05:26:7a:3c ? Broadcast    0x4008 46 Ethernet II
   37 123.177255060 3c:00:05:26:7a:3c ? Broadcast    0xc020 64 Ethernet II
   38 123.177276060 3c:00:05:26:7a:3d ? Broadcast    0x4008 46 Ethernet II
   39 123.177286180 3c:00:05:26:7a:3d ? Broadcast    0xc020 64 Ethernet II
   40 123.183086110 3c:00:05:26:7a:3a ? Broadcast    0x4008 46 Ethernet II
   41 123.183121440 3c:00:05:26:7a:3a ? Broadcast    0xc020 64 Ethernet II
   42 123.188783060 3c:00:05:26:7a:3b ? Broadcast    0x4008 46 Ethernet II
   43 123.188818240 3c:00:05:26:7a:3b ? Broadcast    0xc020 64 Ethernet II
   44 123.194440550 3c:00:05:26:7a:3c ? Broadcast    0x4008 46 Ethernet II
   45 123.194475810 3c:00:05:26:7a:3c ? Broadcast    0xc020 64 Ethernet II
   46 123.200055810 3c:00:05:26:7a:3d ? Broadcast    0x4008 46 Ethernet II
   47 123.200090840 3c:00:05:26:7a:3d ? Broadcast    0xc020 64 Ethernet II
   48 123.205375980 74:00:08:1a:4d:53 ? 3c:00:05:26:7a:3a 0x4020 46 Ethernet II
   49 123.205492860 74:00:08:1a:4d:54 ? 3c:00:05:26:7a:3b 0x4020 46 Ethernet II
   50 123.205606720 74:00:08:1a:4d:55 ? 3c:00:05:26:7a:3c 0x4020 46 Ethernet II
   51 123.205695060 3c:00:05:26:7a:3e ? Broadcast    0x4008 46 Ethernet II
   52 123.205719980 74:00:08:1a:4d:56 ? 3c:00:05:26:7a:3d 0x4020 46 Ethernet II
   53 123.205733180 3c:00:05:26:7a:3e ? Broadcast    0xc020 64 Ethernet II
   54 123.308717700 74:00:08:1a:4d:57 ? 3c:00:05:26:7a:3e 0x4020 46 Ethernet II

Decoding the DSA tags -

0x4008 (pkt len 46)
0100 0000 0000 1000
01 0 00000 => mode 1 (from-cpu)
00001 000 => port 1
[Packet to be sent out of port 1]

0xc020 (pkt len 64)
1100 0000 0010 0000
11 0 00000 => mode 3 (forward)
00100 000 => port 4
[Packet received from port 4]

0x4020 (pkt len 46)
0100 0000 0010 0000
01 0 00000 => mode 1 (from-cpu), b29 unset
00100 000 => port 4
[Packet to be sent out of port 4]

From the DSA tag decodes, we see that packet is sent out of port 1 (lan1), received back from port 4 (lan4) - after padding.

However for the packet sent out of port 4 (lan4), we don't see packets with DSA tags showing received from port 1 (lan1).

Interface stats show reply packets sent out of lan4 increment Tx stats, but Rx stats on lan1 don't increment.

If I repeat the experiment with sending out ARP requests out of lan4 instead of lan1, the same thing happens - replies sent by lan1 are not received by lan4.

This problem is at times not seen, but mostly it is seen. Not able to figure out why.

The problem is NOT seen with the X1, X2 interfaces leading me to beleive that the DSA may be playing a role here. The addition of a DUT between the two 1G ports may also resolve the problem, but I wasn't able to test that due to lack of a suitable DUT at the moment. Addition of DUT does not solve the problem - the LED does blink on the recipient port so the packet is received at the Phy layer, but something after that is dropping the packets.