Parallelize router initialization

[noobHappylife]

Title

Use ProcessPoolExecutor to parallelize set_model_list in Router initialization

Relevant issues

Related to #7647

Type

🆕 New Feature

Changes

Use ProcessPoolExecutor to parallelize set_model_list in Router initialization

[REQUIRED]

A test script to test the router creation time

import time
from litellm import Router

# Function to create a list of model configurations
def generate_model_configs(num_models):
    model_configs = []
    for i in range(num_models):
        model_config = {
            "model_name": f"model_{i}",
            "litellm_params": {
                "model": f"openai/model_{i}",
                "api_key": "your-api-key",
                "api_base": "https://api.openai.com/v1"
            }
        }
        model_configs.append(model_config)
    return model_configs

# Function to test router creation time
def test_router_creation(num_models):
    model_configs = generate_model_configs(num_models)

    start_time = time.time()
    router = Router(model_list=model_configs)
    end_time = time.time()

    creation_time = end_time - start_time
    print(f"Router num models: {len(router.model_list)}")
    print(f"Time to create router with {num_models} models: {creation_time:.4f} seconds")

# Test with varying number of models
model_counts = [10, 20, 30, 40, 50]
for count in model_counts:
    test_router_creation(count)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 9, 2025 7:14am

[krrishdholakia]

Hey @noobHappylife this is a great PR. Can you add some testing for your change.

Possibly in this file -

litellm/tests/local_testing/test_router_client_init.py

Line 26 in 8e2653c

async def test_router_init():

we typically add a unit test for all functions in router.py