SystemMessage Not Effective

[babelpainterwell]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.

Example Code

system_prompt = "Don't speak antyhing."
agent = AgentM(model, tools)

# the initial task input
# messages = [HumanMessage(content="What is score of NBA Final Game 1?")]
messages = [HumanMessage(content="The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? ")]
messages = [SystemMessage(content=system_prompt)] + messages

# Execute the task
config = {"configurable": {"thread_id": "1"}}
try: 
  result = agent.graph.invoke({"messages": messages}, config=config)
  print(result['messages'][-1].content)
  print("*" * 20)
  print(agent.graph.get_state(config))
except UserExitError as e:
  print(e.message)

Error Message and Stack Trace (if applicable)

****************************% python tutorial.py
Please provide your student ID so I can assist you in calculating the total number of students in your class.

---

StateSnapshot(values={'messages': [SystemMessage(content="Don't say antyhing.", id='ffc8061c-796f-430e-9d7f-9c82852e4a12'), HumanMessage(content='The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? ', id='a1e0d963-eda8-46f4-b3c3-3ae26780c08b'), AIMessage(content='Please provide your student ID so I can assist you in calculating the total number of students in your class.', response_metadata={'token_usage': {'completion_tokens': 22, 'prompt_tokens': 150, 'total_tokens': 172}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None}, id='run-f5cdb365-3cc9-4025-80f0-0d80fd2d6d34-0', usage_metadata={'input_tokens': 150, 'output_tokens': 22, 'total_tokens': 172})], 'query_human': False}, next=(), config={'configurable': {'thread_id': '1', 'thread_ts': '1ef26ada-893a-6892-8001-0bd642095671'}}, metadata={'source': 'loop', 'step': 1, 'writes': {'llm': {'messages': [AIMessage(content='Please provide your student ID so I can assist you in calculating the total number of students in your class.', response_metadata={'token_usage': {'completion_tokens': 22, 'prompt_tokens': 150, 'total_tokens': 172}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None}, id='run-f5cdb365-3cc9-4025-80f0-0d80fd2d6d34-0', usage_metadata={'input_tokens': 150, 'output_tokens': 22, 'total_tokens': 172})], 'query_human': False}}}, created_at='2024-06-09T22:14:37.122152+00:00', parent_config={'configurable': {'thread_id': '1', 'thread_ts': '1ef26ada-80c4-6d52-8000-dd00aa835319'}})

Description

• SystemMessage can be correctly appended in the beginning, by printing out graph.get_state(), but it seems the agent can't be guided by any info in the SystemMessage.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6000
Python Version: 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:37:23)
[Clang 13.0.0 (clang-1300.0.29.30)]

Package Information

langchain_core: 0.2.4
langchain: 0.2.2
langchain_community: 0.2.3
langsmith: 0.1.74
langchain_anthropic: 0.1.15
langchain_openai: 0.1.8
langchain_text_splitters: 0.2.1
langgraph: 0.0.64

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langserve

[hinthornw]

Hi @babelpainterwell - thanks for raising this issue. Do you have a code snippet of what your actual graph looks like? Hard to give specific advice without it.

It seems from what you've shared that you're trying to instruct the LLM to never generate a response using the system message. Depending on the model, it's sometimes quite hard to get them actually to not respond (i.e., immediately generate a stop token) - it's sometimes easier to have them generate a canned "end" phrase before it like "NO RESPONSE" or something, though even that is imperfect.

IIRC, this is more of a prompt engineering / machine learning problem that you're running into rather than an issue with the actual LangGraph framework. With these problems, I typically recommend writing tests and then modifying the prompt to pass these tests (i.e., eval-driven development). LangSmith is one such tool that makes this quite convenient.

[hinthornw]

Gonna move this to discussions since at first glance it appears there isn't an actual bug with LangGraph associated with this ticket. Open to being convinced otherwise (can re-open if so)

[babelpainterwell]

Hi @hinthornw, thanks for reaching out. system_prompt = "Don't speak antyhing." is just one of the system prompts I tried. Among other trails, I used for example system_prompt = "You are an AI agent and can answer people's questions.For any interaction with humans, you can ask for additional information using the tool 'RequestAdditionalInfoFromHuman', which should be used ahead of any tool calls.", which doesn't guide the agent to utilize tools for additional information, while when I append the system prompt to the first human message, the agent knows to use the 'RequestAdditionalInfoFromHuman' tool.

EXAMPLE 1
` # create the agent
system_prompt = "You are an AI agent and can answer people's questions.For any interaction with humans, you can ask for additional information using the tool 'RequestAdditionalInfoFromHuman', which should be used ahead of any tool calls."
# system_prompt = "Don't speak antyhing."
agent = AgentM(model, tools)

# the initial task input
# messages = [HumanMessage(content="What is score of NBA Final Game 1?")]
messages = [HumanMessage(content=f"The total number of students in my class is to use my student id and plus 10. So how many students are there in my class?")]
# messages = [HumanMessage(content=f"The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? \n {system_prompt}")]
messages = [SystemMessage(content=system_prompt)] + messages

# Execute the task
try: 
    result = agent.graph.invoke({"messages": messages, "supervisor_mode": True}, config=config)
    # result = agent.graph.invoke({"messages": messages})
    print(result['messages'][-1].content)
    print("*" * 20)
    print(agent.graph.get_state(config))
except UserExitError as e:
    print(e.message)`

EXAMPLE 1 state (note there is no tool call in the first ai message)
{'messages': [SystemMessage(content="You are an AI agent and can answer people's questions.For any interaction with humans, you can ask for additional information using the tool 'RequestAdditionalInfoFromHuman', which should be used ahead of any tool calls.", id='bf53e1b3-4a82-4550-acdb-123a7b77d4cd'), HumanMessage(content='The total number of students in my class is to use my student id and plus 10. So how many students are there in my class?', id='43d758d9-a60f-41ee-a80c-9f2e3696a339'), AIMessage(content='In order to determine the total number of students in your class, I need to know your student ID. Could you please provide your student ID?', response_metadata={'token_usage': {'completion_tokens': 30, 'prompt_tokens': 196, 'total_tokens': 226}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None}, id='run-532c6adb-a338-4891-bef2-c99c32b729a5-0', usage_metadata={'input_tokens': 196, 'output_tokens': 30, 'total_tokens': 226})], 'query_human': False, 'supervisor_mode': True}

EXAMPLE 2
`# create the agent
system_prompt = "You are an AI agent and can answer people's questions.For any interaction with humans, you can ask for additional information using the tool 'RequestAdditionalInfoFromHuman', which should be used ahead of any tool calls."
# system_prompt = "Don't speak antyhing."
agent = AgentM(model, tools)

# the initial task input
# messages = [HumanMessage(content="What is score of NBA Final Game 1?")]
# messages = [HumanMessage(content=f"The total number of students in my class is to use my student id and plus 10. So how many students are there in my class?")]
messages = [HumanMessage(content=f"The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? \n {system_prompt}")]
# messages = [SystemMessage(content=system_prompt)] + messages

# Execute the task
try: 
    result = agent.graph.invoke({"messages": messages, "supervisor_mode": True}, config=config)
    # result = agent.graph.invoke({"messages": messages})
    print(result['messages'][-1].content)
    print("*" * 20)
    print(agent.graph.get_state(config))
except UserExitError as e:
    print(e.message)`

EXAMPLE 2 state (note there is a tool call when we append the system prompt in the first human message)
{'messages': [HumanMessage(content="The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? \n You are an AI agent and can answer people's questions.For any interaction with humans, you can ask for additional information using the tool 'RequestAdditionalInfoFromHuman', which should be used ahead of any tool calls.", id='b4aedc56-46e3-42f2-8ede-780f2d780a8f'), AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_O9nennkpAmnqjAIlLLwWoJQe', 'function': {'arguments': '{"request":"Can you please provide your student ID?"}', 'name': 'RequestAdditionalInfoFromHuman'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 24, 'prompt_tokens': 197, 'total_tokens': 221}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-bcbcc9f6-069f-495c-9763-6042f752999d-0', tool_calls=[{'name': 'RequestAdditionalInfoFromHuman', 'args': {'request': 'Can you please provide your student ID?'}, 'id': 'call_O9nennkpAmnqjAIlLLwWoJQe'}], usage_metadata={'input_tokens': 197, 'output_tokens': 24, 'total_tokens': 221})], 'query_human': True, 'supervisor_mode': True} Can you please provide your student ID? 123 {'messages': [HumanMessage(content="The total number of students in my class is to use my student id and plus 10. So how many students are there in my class? \n You are an AI agent and can answer people's questions.For any interaction with humans, you can ask for additional information using the tool 'RequestAdditionalInfoFromHuman', which should be used ahead of any tool calls.", id='b4aedc56-46e3-42f2-8ede-780f2d780a8f'), AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_O9nennkpAmnqjAIlLLwWoJQe', 'function': {'arguments': '{"request":"Can you please provide your student ID?"}', 'name': 'RequestAdditionalInfoFromHuman'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 24, 'prompt_tokens': 197, 'total_tokens': 221}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-bcbcc9f6-069f-495c-9763-6042f752999d-0', tool_calls=[{'name': 'RequestAdditionalInfoFromHuman', 'args': {'request': 'Can you please provide your student ID?'}, 'id': 'call_O9nennkpAmnqjAIlLLwWoJQe'}], usage_metadata={'input_tokens': 197, 'output_tokens': 24, 'total_tokens': 221}), ToolMessage(content='123', id='dfdc9d08-e6a8-4b1f-aec5-6e38518a492d', tool_call_id='call_O9nennkpAmnqjAIlLLwWoJQe'), AIMessage(content='Given your student ID is 123, the total number of students in your class is:\n\n123 (your student ID) + 10 = 133 students.', response_metadata={'token_usage': {'completion_tokens': 33, 'prompt_tokens': 233, 'total_tokens': 266}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None}, id='run-a5e693e2-512f-44aa-8c13-77ef7f23f205-0', usage_metadata={'input_tokens': 233, 'output_tokens': 33, 'total_tokens': 266})], 'query_human': False, 'supervisor_mode': True}

[jflaflamme]

I have the same issue with using invoke, I have different agents injecting SystemMessage but the final agent do not see any of the them, only HumanMessages, despite all the messages being in the state and displayed in debug mode.

When using a chain rather than an agent function, the system messages are passed correctly (but then raising errors because a Dict is expected rather than Messages List). But that shows the invoke has an issue processing system messages.

So in clear: Does ainvoke process only human messages?

[hinthornw]

ainvoke and invoke process the same data. Do you have linked code? It seems from teh error you shared that one of the sub-graphs you have expects {"messages": <the messages>} and it's being passed a list (<messages>), so you'd have to update your function to put the list in a dict.

[jflaflamme]

ainvoke and invoke process the same data. Do you have linked code? It seems from teh error you shared that one of the sub-graphs you have expects {"messages": <the messages>} and it's being passed a list (<messages>), so you'd have to update your function to put the list in a dict.

My last question should have mentioned invoke instead of ainvoke.

The point was that, when I pass to a chain rather than agent, the SystemMessages are processed correctly.

I am not sure if it is the same issues the OP meant.