AutoGen with Azure AI Foundry CodeLlama managed endpoint #5121
Unanswered
antonslutskyms
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello dear community! I'm trying to use AutoGen with a CodeLlama-70b-Instruct-hf model deployed via Azure AI Foundry managed endpoint and getting the following error message:
APIStatusError: Error code: 424 - {'detail': 'Not Found'}
Looking online, this seems to be related to an issue discussed in:
Azure/azure-sdk-for-python#39025
where adding /scope to the end of the base_url resulted in issues.
I tried to remove /scope from the base_url parameter of the LLM config block but getting the same error.
Tried both 0.2 and 0.4 versions of AugoGen -- same error in both.
My 0.2 version looks like this:
llm_config = { 'config_list': [{
'model': 'codellama',
'api_key': 'my api key',
'base_url': 'https://my endpoint name.my region.inference.ml.azure.com/score', # or "https://my endpoint name.my region.inference.ml.azure.com/" with same result
'api_type': 'azure',
}] }
assistant = AssistantAgent('assistant', llm_config=llm_config,
code_execution_config=False,
max_consecutive_auto_reply=30,
human_input_mode='NEVER'
)
0.4 version is:
az_model_client = OpenAIChatCompletionClient(
model= "codellama",
api_key="my key",
base_url="https://my endpoint name.my region.inference.ml.azure.com/score", # or without /score
model_capabilities={
"vision": False, # Replace with True if the model has vision capabilities.
"function_calling": False, # Replace with True if the model has function calling capabilities.
"json_output": False, # Replace with True if the model has JSON output capabilities.
},
max_tokens=128000
)
agent = AssistantAgent("assistant", az_model_client)
Would anyone have any suggestions?
TIA!
Beta Was this translation helpful? Give feedback.
All reactions