Porting auto-scaling internal code

[rafidka]

Issue #, if available: #46

Description of changes: This PR ports the internal code we have related to auto-scaling. It ensures graceful shutdown of workers in case of auto-scaling.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[ashishgo-aws]

Overall, the change looks fine. Two things though:

1. Where is this logic placed? (console_handler.py#L123-L141)
2. What testing has been done so far related to the autoscaling changes?

[rafidka]

@ashishgo-aws

Overall, the change looks fine. Two things though:

1. Where is this logic placed? https://code.amazon.com/packages/AWSPinwheelImages/blobs/66f9433e722b91e4d01a517c13b5ba23b874dcc2/--/images/airflow/2.8.1/logs/console_handler.py#L123-L141

Ah, I missed this part.

2. What testing has been done so far related to the autoscaling changes?

This is still WIP, so testing is still on-going. I appreciate if you share some testing tips/DAGs that can help with this.

[rafidka]

@ashishgo-aws , I addressed all comments and updated the PR. Please take a look. Thanks!