Upgrading kafka-clients to 3.2.1 causing multiple unittest failures

[suez1224]

We hit this issue in OpenAI production, KAFKA-13840. The kafka-clients version affected are from 2.6.2 to 3.2.1. We tried to upgrade kafka-clients to 3.2.3 and hit a API back incompatibility change in KafkaConsumer constructor and multiple unittest failures after attempting to fix the back incompatibility, e.g.

In DlqTopicKafkaFetcherTest.testBoundedJobs:

Wanted but not invoked:
sink.submit(<any>);
-> at com.uber.data.kafka.datatransfer.worker.common.Sink.submit(Sink.java:21)

However, there was exactly 1 interaction with this mock:
sink.isRunning();
-> at com.uber.data.kafka.datatransfer.worker.fetchers.kafka.AbstractKafkaFetcherThread.doWork(AbstractKafkaFetcherThread.java:240)


	at app//com.uber.data.kafka.datatransfer.worker.common.Sink.submit(Sink.java:21)
	at app//com.uber.data.kafka.consumerproxy.worker.fetcher.DlqTopicKafkaFetcherTest.testBoundedJobs(DlqTopicKafkaFetcherTest.java:297)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:566)

Current suspect is the major version change in kafka-clients from 2.x.x to 3.x.x has some internal behavior changes that break uForwarder's assumption on Kafka client behavior. Downngrading the kafka-clients to 2.4.1 and 2.6.1 both succeed in all unittests.