Methods for optimizing inference latency and cost in prod?

[FaizShah]

Hi all,

I am in the early stages of an app involving TTS and have been looking for methods to optimize the cost and latency of tortoise inference in prod.

I am curious to know if anyone else has tried to deploy tortoise in prod and what methods have you successfully used?

So far I am going to try:

• Quantizing some of the models in the pipeline
• Swapping GroupNorm for BatchNorm and using fuse_modules
• CPU inference with OpenVINO (possibly lower cost than spot instance k80s)

I found that using the ultra_fast preset is good enough to produce TTS that’s almost at parity with GCP TTS but at much lower cost. I’m still experimenting with fiddling with the params but so far I haven’t found anything that can significantly decrease inference latency while maintaining the same quality.

Curious to hear any suggestions you have as I’m still new to the DL/transformers space?

Thanks,
Faiz

[mvoodarla]

Hey, thanks for your work on this. Curious to hear how the optimizations went and at what speed you're able to do TTS at!