Facing difficulty using Gemini Flash 2.0 with function calling

[gvijqb]

What happened?

Whenever there's a function call with Gemini Flash 2.0 in autogen, I get this error:
Google GenAI exception occurred while calling Gemini API: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties: should be non-empty for OBJECT type

What did you expect to happen?

It should be able to use the tool call properly.

How can we reproduce it (as minimally and precisely as possible)?

Just an assistantagent with a function call should be able to reproduce it.

AutoGen version

0.2

Which package was this bug in

Core

Model used

Gemini flash 2.0 exp

Python version

3.11

Operating system

Ubuntu 22.04

Any additional info you think would be helpful for fixing this bug

No response

[ekzhu]

Gemini has added support for openai style endpoint, can you try that without specifying the api_type?

[gvijqb]

@ekzhu If I don't specify api_type then I am getting authentication error:

raise self._make_status_error_from_response(err.response) from None                                                                          
openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: AIza********************************VQ. You can
 find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

I guess it is expecting it to be a openAI API key when I do not specify api_type as google.

I am using OAI Config List method like this:

OAI_CONFIG_FILE:

[
    {
        "model":"gemini-2.0-flash-exp",
        "api_key": "KEY",
        "api_type": "google"
    }
]

Runtime:

config_list_gemini_flash = autogen.config_list_from_json(
    "OAI_CONFIG_LIST",
    filter_dict={
        "model": ["gemini-2.0-flash-exp"]
    }
)

gemini_flash_config = {
    "cache_seed": 42,
    "max_retries": 5,
    "config_list": config_list_gemini_flash,
    "timeout": 30000,
}

Is there any configuration issue in this workflow?

[ekzhu]

Based on https://ai.google.dev/gemini-api/docs/openai#:~:text=Gemini%20models%20are%20accessible%20using%20the%20OpenAI%20libraries,recommend%20that%20you%20call%20the%20Gemini%20API%20directly.

You need to set base_url field to: "https://generativelanguage.googleapis.com/v1beta/openai/", and also the API key to your Gemini's key.