Managing Large Tool Call Outputs in LangGraph

[developer-hassan]

Context:

I am currently working on a persistence application using Python and LangGraph where the LLM decides whether to invoke a tool call based on the context of the user's input. If there is no tool call, I stream the output using FastAPI's StreamingResponse. However, when a tool call is made, the result is a large JSON payload (approximately 3000-4000 lines per call) since it involves API responses from third-party services like fetching the latest news articles, large shopping results, or comprehensive food store rankings.

Throughout a single chat session, a user might trigger multiple tool calls (up to 10 or more). If each tool call output were appended to the session memory ("messages" payload), the memory size would quickly become unmanageable, significantly impacting performance and scalability.

Current Approach:

At the moment, I am skipping the appending of tool call outputs to the memory messages. This prevents the memory from ballooning to an unmanageable size but creates challenges when it comes to maintaining context or history related to these tool calls and LLM is not aware of what responses it has returned me from my query.

Questions for the Community:

1. Is skipping the tool call outputs the best approach, or are there alternative strategies to handle such scenarios?
2. How can we maintain the context or history of these tool calls without overloading the memory?
   • For example, could a summarized or compressed version of the tool call output be a viable option?
3. Are there existing patterns or best practices in LangGraph for managing large payloads in persistent applications?
4. Would external storage (like a database or caching system) be a recommended solution for persisting these large outputs and referencing them as needed without keeping them in memory?

Additional Considerations:

• The tool call outputs are critical for the application, as they often provide the key information that the user needs.
• If the outputs are stored externally, what would be the best way to reference or retrieve them efficiently during a session?
• How might this approach scale with concurrent users generating multiple tool calls?

I would appreciate insights, recommendations, or examples from the community to help address this challenge. Thank you!

[developer-hassan]

Any updates?