Agent is not interpreting tool message, instead going into endless loop

[manojmanivannan]

I followed the customer support example of langgraph, and created a simple tool calling agent using OllamaFunctions. Agent understands the question, calls the right tool, however, it goes into endless loop. The message from the tool is not understood as the final answer. Maybe I did something stupid, please point it out. Any help is appreciated.

My notebook is here: https://github.com/manojmanivannan/agent_rag_tools/blob/example/ask_main.ipynb

from langgraph.checkpoint.sqlite import SqliteSaver
from langgraph.graph import END, StateGraph, START
from langgraph.prebuilt import tools_condition

def should_continue(state):
    messages = state["messages"]
    last_message = messages[-1]
    print('Last Message at Should continue', last_message)
    # If there is no function call, then we finish
    if not last_message.tool_calls:
        return "end"
    # Otherwise we continue
    else:
        return "continue"
    
builder = StateGraph(State)


# Define nodes: these do the work
builder.add_node("assistant", Assistant(part_1_assistant_runnable))
builder.add_node("tools", create_tool_node_with_fallback(part_1_tools))
# Define edges: these determine how the control flow moves
builder.add_edge(START, "assistant")
builder.add_conditional_edges(
    "assistant",
    tools_condition,
    # should_continue,
    # {
    #     "continue":"tools",
    #     "end": END
    # }
)
builder.add_edge("tools", "assistant")

# The checkpointer lets the graph persist its state
# this is a complete memory for the entire graph.
memory = SqliteSaver.from_conn_string(":memory:")
part_1_graph = builder.compile(checkpointer=memory)

[hinthornw]

One thing that looks incorrect above is you've commented out the path map but are returning "end", which isn't a node in the graph (did you mean the dunder "__ end __''" )

If that's not it, What model are you using? Do you have a langsmith trace to share?

The cause of this could be anything from "the model is bad" (language models used to have the repetition failure mode more often) to "the message isn't being added to the state"

[manojmanivannan]

Thanks @hinthornw for your response.
Im using the buildin tool condition check using tools_condition as per this example https://langchain-ai.github.io/langgraph/tutorials/customer-support/customer-support/#define-graph

you can see this trace: https://smith.langchain.com/public/289988a4-9b97-4686-9c6d-448ea97730eb/r

[manojmanivannan]

im using llama3:8B and mistral

[hwchase17]

It may just be prompting issues? Have you tried clarifying that it doesnt need to call tools and when to return direct?

[manojmanivannan]

I got it to stop infinite loop (sometime it still does loop), but not sure if my solution was right. Basically, i made the tool respond like {"response": "ANSWER_FROM_TOOL"}. And also ollama_functions.py has its own default system prompts for the tool and response.
But im hitting another issue. Whenever it does not loop, the response from the model, is not parse-able to ollama_function.

File ~/Github/agent_rag_tools/venv/lib/python3.9/site-packages/langchain_experimental/llms/ollama_functions.py:395, in OllamaFunctions._generate(self, messages, stop, run_manager, **kwargs)
    393         response = parsed_chat_result["response"]
    394     else:
--> 395         raise ValueError(
    396             f"Failed to parse a response from {self.model} output: "
    397             f"{chat_generation_content}"
    398         )
    399     return ChatResult(
    400         generations=[
    401             ChatGeneration(
   (...)
    406         ]
    407     )
    409 called_tool_arguments = (
    410     parsed_chat_result["tool_input"]
    411     if "tool_input" in parsed_chat_result
    412     else {}
    413 )

ValueError: Failed to parse a response from llama3 output: {}

This is my prompt, btw,

ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are a helpful customer support assistant for interacting with database "
            " Use the provided tools to search inside the database "
            # " When searching, be persistent. Expand your query bounds if the first search returns no results. "
            " If a search comes up empty, expand your search before giving up."
            " If you have a response from a tool that answers the question, generate the final answer"
            # "\nCurrent time: {time}.",
        ),
        ("placeholder", "{messages}"),
    ]
).partial(time=datetime.now())

Here is langsmith trace: https://smith.langchain.com/public/32ef6e69-2a4e-4410-928f-31987def6973/r
Notebook: https://github.com/manojmanivannan/agent_rag_tools/blob/example/ask_main.ipynb

[manojmanivannan]

It turns out the problem was my model (llama3) was not good tool call or my prompt.
However, using the langchain-ollama (chatOllama) instance with llama3-groq-tool-use model, the graph works.
https://github.com/manojmanivannan/agent_rag_tools/blob/example/ask_main.ipynb