Streaming with ReAct Agent + Ollama does not work?

[ahenkes1]

Hello everyone. I have built a ReAct agent using the Llama3.2 model from Ollama. I used
langgraph.prebuilt.create_react_agent and langchain_ollama.ChatOllama. I created some tools and added them to the model.

For pretty printing, I am using the following code to astream over the agent:

        async for chk in agent.astream(prompt, config=config):  # type: ignore
            result = list(chk.values())[0]
            try:
                message = result["messages"][-1]
                role = message.response_metadata["message"].role
                tools = message.tool_calls

                if role == "assistant" and tools == []:
                    result = "AI: " + message.content
                    formatted_string = textwrap.fill(result, width=79)
                    print(formatted_string)
                    print(f"{79 * '-'}\n")

            except Exception:
                pass

This works fine, but the behavior is odd. There is no per token streaming, but instead the whole answer is plotted. I recall that I read somewhere, that tooling and streaming is incompatible in langgraph. Can someone make sense out of this?

[ahenkes1]

Hello everyone. I have built a ReAct agent using the Llama3.2 model from Ollama. I used
langgraph.prebuilt.create_react_agent and langchain_ollama.ChatOllama. I created some tools and added them to the model.

For pretty printing, I am using the following code to astream over the agent:

        async for chk in agent.astream(prompt, config=config):  # type: ignore
            result = list(chk.values())[0]
            try:
                message = result["messages"][-1]
                role = message.response_metadata["message"].role
                tools = message.tool_calls

                if role == "assistant" and tools == []:
                    result = "AI: " + message.content
                    formatted_string = textwrap.fill(result, width=79)
                    print(formatted_string)
                    print(f"{79 * '-'}\n")

            except Exception:
                pass

This works fine, but the behavior is odd. There is no per token streaming, but instead the whole answer is plotted. I recall that I read somewhere, that tooling and streaming is incompatible in langgraph. Can someone make sense out of this?

Additionally, other Ollama model do not work either...

[vbarda]

responded on this issue #3259 as well -- you need to use stream_mode="messages" in .stream() / .astream() method

[ahenkes1]

Thank you very much!