Try to run 2 Ollam

[dzm1098]

Thank you for your amazing work. I am trying to use two different OLLama models locally as strong (Phi4) and weak (Llama3 8b). I can run both models using method A which is also known as python sdk. But I want to try using method B.My method B never worked.

This is my input
python -m routellm.openai_server --routers mf --config config.example.yaml --base-url http://localhost:11434/v1/ --api-key ollama --strong-model phi4 --weak-model llama3

Also I change the router chat
client = OpenAI( api_key=openai_api_key, base_url=openai_api_base, )
When I run it in CMD it shows
INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)

After that I using python -m examples.router_chat --router mf --threshold 0.11593 to start the server
However when I input anything in server it shows
`Provider List: https://docs.litellm.ai/docs/providers

INFO: 127.0.0.1:53951 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\uvicorn\middleware\proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\fastapi\applications.py", line 1054, in call
await super().call(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\middleware\errors.py", line 187, in call
raise exc
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\middleware\errors.py", line 165, in call
await self.app(scope, receive, _send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\middleware\exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app
raise exc
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\routing.py", line 735, in app
await route.handle(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\routing.py", line 288, in handle
await self.app(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette_exception_handler.py", line 53, in wrapped_app
raise exc
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\starlette\routing.py", line 73, in app
response = await f(request)
^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\fastapi\routing.py", line 301, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\fastapi\routing.py", line 212, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\OneDrive\大学\CSC490\RouteLLM-main\routellm\openai_server.py", line 124, in create_chat_completion
res = await CONTROLLER.acompletion(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\OneDrive\大学\CSC490\RouteLLM-main\routellm\controller.py", line 170, in acompletion
return await acompletion(api_base=self.api_base, api_key=self.api_key, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\litellm\utils.py", line 1241, in wrapper_async
raise e
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\litellm\utils.py", line 1093, in wrapper_async
result = await original_function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\litellm\main.py", line 426, in acompletion
_, custom_llm_provider, _, _ = get_llm_provider(
^^^^^^^^^^^^^^^^^
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\litellm\litellm_core_utils\get_llm_provider_logic.py", line 356, in get_llm_provider
raise e
File "C:\Users\dzm10\AppData\Local\Programs\Python\Python312\Lib\site-packages\litellm\litellm_core_utils\get_llm_provider_logic.py", line 333, in get_llm_provider
raise litellm.exceptions.BadRequestError( # type: ignore
litellm.exceptions.BadRequestError: litellm.BadRequestError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. You passed model=llama3
Pass model as E.g. For 'Huggingface' inference endpoints pass in completion(model='huggingface/starcoder',..) Learn more: https://docs.litellm.ai/docs/providers`

But at the same time, even if I follow the readme and put gbt4 and llama3 together, it doesn't work. Do you have any solutions? Thanks