Implicit (split thread) MemGPT

[cpacker]

Add implicit version of MemGPT with two separate threads (dialogue thread + memory management thread)

[chandeldivyam]

Discord Discussion

- create new agent class
- agent.step now runs one thread (can itself be an agent) that does inner monologue + pause heartbeats
- and another thread that does all the other memory functions
- you can run these async, but the entire high-level step() should probably (?) be blocking
- to split the threads, you also have to split the system prompt into 2, and probably provide some in-context examples for the memory functions thread (might not be needed)

• Looking into it, and will come back with a proposed solution.

Also, it would be helpful to understand more about Why this is necessary. Thanks!

[chandeldivyam]

my understanding

Current Framework on how we are handling requests:

The above-mentioned proposal might not directly work as memory functions have a direct dependency on inner monologue (i.e.) the response received from the LLM.

What we can do to optimize it better is, split the update functions and retrieval functions to different agents. As, when the agent have to retrieve some information, it has a need to send heartbeat_request as true. But, there is no need for another request to LLM, if we are able separate the update functions.

Here, we can send the new message using send_message function + parallel execution of update memory

@cpacker I would love to understand your perspective. Also, would want to understand what is the goal for performing this. What we are saying is handle_ai_response breaks away from the step function. But, it is blocked on get_ai_reply_async

The memory functions are blocked by the first request as they are dependent.

I do not have a holistic understanding of the project and would like to know your thoughts!

[HaileyStorm]

One advantage I see to this is the eventually we might want the memory management agent to (intelligently, with some provided contextual information) modify the retrieved information such as filtering out most relevant parts or summarizing due to length. Would make it easier to use third party tools for those operations too.

[HaileyStorm]

Also, you can probably use a cheaper model for the memory management thread.