tpu-client-next: add mechanism to implement custom scheduler

[KirillLykov]

Problem

The current scheduler implementation works for SendTransactionService or similar applications where the client checks if the transaction has been added to the block and retries when necessary. But for some other applications, like transaction-bench client, we want to have a custom strategy. For example, a combination of try_send and send to some future leaders.

This PR introduces trait that implements most of the logic except one method send_to_workers, which should be implemented by user.

For the sake of backport simplicity, should be added after #4454

Summary of Changes

[ilya-bobyr]

Handing errors in async code is a bit tricky.

Here you are saying that if send_to_workers() fails, you are just going to return the error to the caller.

Just want to make sure that this is correct behavior.
Considering that under normal flow there are these calls, that are executed when the worker scheduler is terminating:

        workers.shutdown().await;
        endpoint.close(0u32.into(), b"Closing connection");
        leader_updater.stop().await;

Are you sure they should not be executed if send_to_workers() fails for any reason?

If not, you may want to store an error, terminate the loop, run the shutdown code and only then return the error to the caller.

[KirillLykov]

Good catch! Thanks

[KirillLykov]

Maybe the best strategy would be to break the loop in this case. Add this to the new trait documentation.

[KirillLykov]

@ilya-bobyr what about this part? I added a last_error which holds the value of the error if it happen during send_to_workers. Added also to the trait documentation clarification that send_to_workers errors are critical meaning that they stop the scheduler.

[ilya-bobyr]

Looks good.
I would probably call it send_to_workers_err or something like that.
I understand that last in the name comes from the fact that you are calling the send function in the loop, and you are talking about the fact that it is the error observed on the last loop iteration.
But at first it was not clear to me, what the name means.