Problem with bpf_redirect_map from userspace

[bewing]

I am trying to better understand XDP, and tried to create a simple example using bpf_redirect_map to redirect traffic from one network namespace to another controlled by a DEVMAP that I manipulate from userspace. I detailed my approach at https://github.com/bewing/xdp-redirect-map

I have confirmed that traffic is being redirected (I don't see anything when using tcpdump on the main NS veth interface), but I can't tell where it's going -- it should be mapped to the other outside veth interface, and be delivered into the other network namespace, but tcpdump there doesn't show the traffic arriving.

Am I making a mistake in index/interface selection or DEVMAP population? Or am I running into a limitation in XDP as it relates to veth drivers? Kernel version is 5.15.0-52-generic

FWIW, trying to insert bits.ReverseByte32(uint32(iface.Index)) into the devmap results in Could not insert into map: update: invalid argument at runtime

[ti-mo]

Hey @bewing, this doesn't look related to anything in the library specifically, but I'll try to give some pointers because everything's so clearly presented. :)

When dealing with undocumented bpf features (as most of them are), the kernel selftests usually contain an example use case or two that can provide some clarity. In the case of devmap, there's https://elixir.bootlin.com/linux/latest/source/samples/bpf/xdp_redirect_map.bpf.c (and xdp_redirect_map_user.c for the userspace counterpart)

The way you populate the map from userspace seems fine, there's not really a possibility for mismatching byte order (ifindex is pulled from the kernel, processed in a native Go program and written back to a map in the same kernel), and I'm not sure what bits.ReverseByte32() is or what it would accomplish.

When dealing with XDP, it's important to keep in mind you're essentially moving raw packet data (including headers) from one interface buffer to the other, bypassing the network stack in doing so, so you're fully responsible for getting that packet into a state in which it 'makes sense' for it to arrive in the destination netns. In this case, I'd suspect the destination MAC (and IP?) of the packet are set to that of an interface in the host namespace, and when XDP makes it magically appear in the receive queue of the target veth and kicks off packet delivery, the network stack has no idea what to do with the packet, meaning it will be dropped/freed.

Look into reverse-path filtering (Linux rp_filter), check dmesg for any warnings (this kind of network scenario is not really expected outside of XDP and should be investigated), and use the venerable cilium/pwru to get a better idea of where your packet ends up.

Hope this helps!

[bewing]

Thanks for your reply! I grabbed the kernel source and confirmed a few things from the sample code. I was able to get the sample working in a straightforward way. Then I tore down and re-built my project code from scratch, and it just worked, with what appears to be no major changes. /shrug

I will note that if you are trying to use link.XDPDriverMode (non-generic) attach on a veth interface, you MUST have an XDP program of some port loaded on the pair of the veth that the packet is directed to (IE, if you direct from foo to bar in the main NS, the eth0 in the barns namespace will need an XDP program on it, even if it's just returns a simple XDP_PASS). Attaching with link.XDPGenericMode does not have this limitation.

I've updated my sample repo to reflect this with a new commit.

[ti-mo]

@bewing Re: XDP driver mode: I'm not sure if that's intended, this might be a bug in the veth driver. I'm told this functionality isn't so widely used. Might be worth a quick email to the mailing list.

[bewing]

Based on the information presented in https://youtu.be/q3gjNe6LKDI?t=937 (and the associated paper https://www.files.netdevconf.info/f/a63b274e50f943a0a474/ at sec 3.2), this is more an undesired current limitation than a bug.