prov/efa: set ep_attr->max_msg_size according to hints

[wenduwan]

ep_attr->max_msg_size is currently set to UINT64_MAX. As a result the application cannot enable zero-copy recv.

[wenduwan]

bot:aws:retest

[shijin-aws]

is it a regression? I don't know if we ever set it correctly, does that mean we never support zero-copy receive?

[wenduwan]

It is a regression. I haven't bisected git history yet but zcpy recv worked on 1.18.x last time I checked.

[shijin-aws]

how was max_msg_size parsed in v1.18.x?

[shijin-aws]

and also remind me how this impact zero copy receive?

[wenduwan]

It is a regression.

I take that back. It has always been there, but it didn't cause problem because the only zcpy use case is doing odd stuff around fi_getinfo https://github.com/aws/aws-cdi-sdk/blob/47a3063218c4af43d909a34e703f18b922bc853a/src/cdi/adapter_efa.c#L365-L378 (maybe that's because they know it's a libfabric bug).

The application modifies fi_info object after calling fi_getinfo, which I think is illegal.

max_msg_size is used to determine if zero-copy receive can be safely used in efa_rdm_ep_set_use_zcpy_rx

I found this bug while trying to add a fabtest to cover the code path.

[shijin-aws]

I am not sure max_msg_size is used in this way. Look at fi_alter_ep_attr(), max_msg_size is not updated from the one in hints.

I think it is an attribute that provider returns to the application in info that indicates the max msg size and the application must comply with that. It's not something that application should customize.

[shijin-aws]

The application modifies fi_info object after calling fi_getinfo, which I think is illegal.

Right, modifying info after fi_getinfo is illegal.

[wenduwan]

I think it is an attribute that provider returns to the application in info that indicates the max msg size and the application must comply with that. It's not something that application should customize.

Hmmmmmmm let me poke the awesome @shefty :

It's indeed ambiguous from the doc whether fi_info->ep_attr->max_msg_size is read-only or can be set by the application. Could you help clarify?

[shefty]

It's not something the app can change. The app can request a specific value when calling fi_getinfo, and if the provider can't support that size, it can fail the request. But the expectation is that this value maps directly to the transport limitation.

[wenduwan]

@shefty Thanks for chiming in. Cool in this case I think the application in question is abusing/misusing the attribute for EFA provider.

[shijin-aws]

Reopen this PR, as further talking with @shefty shows that provider can choose to use a value of max_msg_size that is equal to what application requests in hints.

[wenduwan]

Rebased branch and running AWS CI