Python: Autofunction calling is hallucinating the end result output for Semantic kernel Orchestrator Example for Python

[rakshahulle]

I have multiple plugins to invoke in sequence for Orchestration purposes. I have added all plugins to Kernel, I am using the automatic function calling to invoke a sequence of prompts for Orchestration flow. But the output for each time is hallucinating. I want to know how to give accurate prompts for Orchestration flow in Python.

[evchaki]

@rakshahulle - here are some samples - https://learn.microsoft.com/en-us/semantic-kernel/concepts/plugins/?pivots=programming-language-python

What model are you using?

[rakshahulle]

I am using Azure OpenAI GPT-4o Model

…

On Thu, Jan 2, 2025 at 9:55 PM EvanChaki_MSFT ***@***.***> wrote: @rakshahulle <https://github.com/rakshahulle> - here are some samples - https://learn.microsoft.com/en-us/semantic-kernel/concepts/plugins/?pivots=programming-language-python What model are you using? — Reply to this email directly, view it on GitHub <#10039 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXCZPRLPJ3O26QPMJBLIGW32IVR67AVCNFSM6AAAAABUMF6A52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRYGAZTMNRTG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[evchaki]

@eavanvalkenburg / @moonbox3 fyi

[alliscode]

Hi @rakshahulle, if you are able to share your code and/or prompts then we might be able to provide some suggestions to reduce hallucinations. You can also take a look at some best practices for writing prompts: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/15-tips-to-become-a-better-prompt-engineer-for-generative-ai/3882935

[rakshahulle]

Hi, I want to create one orchestration flow where the sequence is important. This is my prompt my use case: Cooking Recipe Recommendation System Step 1: Classify the image Classify the image into one of the following food categories: "Vegetarian" or "Non-Vegetarian." Step 2: Search Recipes According to the classification output, search for the top 10 {{$output}} recipes on Bing Web. Step 3: Retrieve Recipe Details For each recipe, retrieve the following details: - id: Recipe ID (unique identifier for the recipe) - name: Name of the recipe - ingredients: List of ingredients required - instructions: Step-by-step cooking instructions - url: URL of the recipe page - image_url: URL of the recipe image Step 4: Store Details into a Database Use a database table to store the recipe details with fields for id, name, ingredients, instructions, url, and image_url. Step 5: Show All Recipes from the Database Display all stored recipes from the database. Step 6: Display First Recipe Highlight the first recipe, showing its name, ingredients, instructions, and image_url.

…

On Thu, Jan 2, 2025 at 11:23 PM Ben Thomas ***@***.***> wrote: Hi @rakshahulle <https://github.com/rakshahulle>, if you are able to share your code and/or prompts then we might be able to provide some suggestions to reduce hallucinations. You can also take a look at some best practices for writing prompts: https://techcommunity.microsoft.com/blog/azure-ai-services-blog/15-tips-to-become-a-better-prompt-engineer-for-generative-ai/3882935 — Reply to this email directly, view it on GitHub <#10039 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXCZPRLAGMQZX5IAH35DNMT2IV4H5AVCNFSM6AAAAABUMF6A52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRYGE2TAMBWGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rakshahulle]

These are some details. I want to use Automatic function calling for this use case. I have 3 plugins 1st is Bing web search, 2nd to retrieve details and store in id, name, instructions, URL, and image_url format, 3rd to store this in db, retrieve these details, display. On Fri, Jan 3, 2025 at 12:04 AM Raksha Hulle ***@***.***> wrote:

…

Hi, I want to create one orchestration flow where the sequence is important. This is my prompt my use case: Cooking Recipe Recommendation System Step 1: Classify the image Classify the image into one of the following food categories: "Vegetarian" or "Non-Vegetarian." Step 2: Search Recipes According to the classification output, search for the top 10 {{$output}} recipes on Bing Web. Step 3: Retrieve Recipe Details For each recipe, retrieve the following details: - id: Recipe ID (unique identifier for the recipe) - name: Name of the recipe - ingredients: List of ingredients required - instructions: Step-by-step cooking instructions - url: URL of the recipe page - image_url: URL of the recipe image Step 4: Store Details into a Database Use a database table to store the recipe details with fields for id, name, ingredients, instructions, url, and image_url. Step 5: Show All Recipes from the Database Display all stored recipes from the database. Step 6: Display First Recipe Highlight the first recipe, showing its name, ingredients, instructions, and image_url. On Thu, Jan 2, 2025 at 11:23 PM Ben Thomas ***@***.***> wrote: > Hi @rakshahulle <https://github.com/rakshahulle>, if you are able to > share your code and/or prompts then we might be able to provide some > suggestions to reduce hallucinations. You can also take a look at some best > practices for writing prompts: > https://techcommunity.microsoft.com/blog/azure-ai-services-blog/15-tips-to-become-a-better-prompt-engineer-for-generative-ai/3882935 > > — > Reply to this email directly, view it on GitHub > <#10039 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AXCZPRLAGMQZX5IAH35DNMT2IV4H5AVCNFSM6AAAAABUMF6A52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRYGE2TAMBWGA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[rakshahulle]

Can you please help me

[moonbox3]

Hello, I’ve been trying to follow the thread here and it isn’t clear to me what the hallucinations are. Is it that your sequence isn’t being followed? If you need a repeatable sequence of events, and simply telling the model how it should sequence things isn’t working, have you looked at our process framework?

[rakshahulle]

Yes, my sequence isn’t being followed. Sometimes it will follow some times it will skip some steps

…

On Thu, Jan 9, 2025 at 5:35 PM Evan Mattson ***@***.***> wrote: Hello, I’ve been trying to follow the thread here and it isn’t clear to me what the hallucinations are. Is it that your sequence isn’t being followed? If you need a repeatable sequence of events, and simply telling the model how it should sequence things isn’t working, have you looked at our process framework? — Reply to this email directly, view it on GitHub <#10039 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXCZPRJVUCQ6UAZOAI3ZXD32JZQZ5AVCNFSM6AAAAABUMF6A52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZZHE4TGMZWGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[moonbox3]

@rakshahulle there are a few ways to go about this given that you need the orchestration to follow a specific flow:

1. You can try really hard to prompt your way to your desired goal. You will need to give clear directions to the model, and hope it is able to complete the plan that you have provided it. It could be something like this (this is just an idea, I wouldn't recommend trying it exactly as is):

<message role="system">
You are an expert at generating plans from a given GOAL. Think step by step and determine a plan to satisfy the specified GOAL using only the FUNCTIONS provided to you. You can also make use of your own knowledge while forming an answer but you must not use functions that are not provided.

[FUNCTIONS]

{{$available_functions}}

[END FUNCTIONS]

Each time you are prompted, follow these steps in this exact order

- Step 1: Classify the image Classify the image into one of the following food categories: "Vegetarian" or "Non-Vegetarian." 
- Step 2: Search Recipes According to the classification output, search for the top 10 {{$output}} recipes on Bing Web. 
- Step 3: Retrieve Recipe Details For each recipe, retrieve the following details: - id: Recipe ID (unique identifier for the recipe) - name: Name of the recipe - ingredients: List of ingredients required - instructions: Step-by-step cooking instructions - url: URL of the recipe page - image_url: URL of the recipe image 
- Step 4: Store Details into a Database Use a database table to store the recipe details with fields for id, name, ingredients, instructions, url, and image_url. 
- Step 5: Show All Recipes from the Database Display all stored recipes from the database. 
- Step 6: Display First Recipe Highlight the first recipe, showing its name, ingredients, instructions, and image_url.
</message>
<message role="user">{{$goal}}</message>

2. Other than trying to prompt your way out, you can have a look at our process framework which is designed to make sure you follow each specified step in the exact order. https://github.com/microsoft/semantic-kernel/tree/main/python/samples/getting_started_with_processes