Switch to untagged send/recv and remove SAS ordering requirement in the RDMA protocol

[AmedeoSapio]

Switching to using untagged send/recv operations in the RDMA protocol. Tags were used for connection request and response, while control messages and eager messages used tag 0. The data previously present in the tag has been moved to the message itself.
In addition, the control message is now using send() instead of senddata(), and the data is carried directly in the message.

Switching to untagged ops means that libfabric won't pre-select the message provided to the recv, so all SENDs are received into bounce buffers now.

This PR also removes the Send-After-Send ordering requirement from libfabric.
The only case where out-of-order sends can be a problem is if a receiver sends a connect response followed by a control message, and they arrive in the opposite order on the sender side. To deal with this rare case, we add a list of control messages in the send communicator to store those early control messages, and process them after the connect response is received.

This also fixes a bug in the accept() error path, that caused accept() to return "success" even in case of errors.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[rauteric]

Thinking about this some more, I don't think the early ctrl special case is needed.

In function handle_ctrl_recv, the case where the ctrl message comes before isend() is really simple: link. We just add the ctrl message to the message buffer and return. Even if the send comm hasn't yet received the connection response, the message buffer will still be initialized (we do this in create_send_comm before we even send the connect message: link).

So I think we should just handle the early ctrl the same way we handle all cases where the receiver starts before sender, and not have this special case.

[rauteric]

High level nit: there are still a few places in rdma.c that refer to "tag" (besides the arguments to send and recv, which refers to the multi-recv tag rather than the Libfabric tag).

[AmedeoSapio]

@rauteric I tested it and confirmed that you are right, if the control message is received before connect response, the control message is added to the msgbuf and retrieved when the isend is called. So there is really no need to do anything new to deal with out-of-order packets. So I will add the commit that removes SAS here as well.