vllm null coordinates

[nivibilla]

Im using the full bf16 model via vllm server. It says these coordinates and then does nothing. On mac

[nivibilla]

raw output dump:

User: You are a GUI agent. You are given a task and your action history, with screenshots. You need to perform the next action to complete the task.

Output Format

Thought: ...
Action: ...

Action Space

click(start_box='[x1, y1, x2, y2]')
left_double(start_box='[x1, y1, x2, y2]')
right_single(start_box='[x1, y1, x2, y2]')
drag(start_box='[x1, y1, x2, y2]', end_box='[x3, y3, x4, y4]')
hotkey(key='')
type(content='') #If you want to submit your input, use "\n" at the end of content.
scroll(start_box='[x1, y1, x2, y2]', direction='down or up or right or left')
wait() #Sleep for 5s and take a screenshot to check for any changes.
finished()
call_user() # Submit the task and call the user when the task is unsolvable, or when you need the user's help.

Note

• Use English in Thought part.
• Write a small plan and finally summarize your next action (with its target element) in one sentence in Thought part.

User Instruction

look for the cheapest 4 star hotel in seattle start from 1st feb for 2 weeks<|im_end|>
Assistant: Thought: To find the cheapest 4-star hotel in Seattle starting from February 1st for 2 weeks, I need to begin by entering the destination in the "Where to?" field. This is the logical first step in narrowing down the search criteria for the desired hotel.
Click on the "Where to?" field to begin entering the destination.
Action: click(start_box='(162,276)')<|im_end|>
Assistant: Thought: To proceed with the task of finding the cheapest 4-star hotel in Seattle, I need to type "Seattle" into the "Going to" field. This will set the destination for the search and allow me to move forward with specifying the dates and other criteria.
Type "Seattle" into the "Going to" field to set the destination for the hotel search.
Action: type(content='seattle')<|im_end|>
User: <|vision_start|><|image_pad|><|vision_end|><|im_end|>
Assistant: Thought: To proceed with the task of finding the cheapest 4-star hotel in Seattle, I need to select the most relevant option from the dropdown menu. The first option, "Seattle, Washington, United States of America," is the most appropriate choice as it represents the general destination for the search.
Click on the "Seattle, Washington, United States of America" option in the dropdown menu to confirm the destination.
Action: click(start_box='(184,344)')<|im_end|>
User: <|vision_start|><|image_pad|><|vision_end|><|im_end|>
Assistant: Thought: To proceed with the task of finding the cheapest 4-star hotel in Seattle, I need to update the dates to reflect the specified range of February 1st for 2 weeks. Clicking on the "Dates" field will allow me to modify the current selection and input the correct dates for the search.
Click on the "Dates" field to begin updating the search dates to February 1st for 2 weeks.
Action: click(start_box='='
(start_box='(353,276)')<|im_end|>
User: <|vision_start|><|image_pad|><|vision_end|><|im_end|>
Assistant: Thought: To proceed with the task of finding the cheapest 4-star hotel in Seattle, I need to update the dates to reflect the specified range of February 1st for 2 weeks. Clicking on the "Dates" field will allow me to modify the current selection and input the correct dates for the search.
Click on the "Dates" field to begin updating the search dates to February 1st for 2 weeks.
Action: click(start_box='='
(start_box='(315,282)')<|im_end|>
User: <|vision_start|><|image_pad|><|vision_end|><|im_end|>
Assistant: Thought: To proceed with the task of finding the cheapest 4-star hotel in Seattle, I need to update the dates to reflect the specified range of February 1st for 2 weeks. Clicking on the "Dates" field will allow me to modify the current selection and input the correct dates for the search.
Click on the "Dates" field to begin updating the search dates to February 1st for 2 weeks.
Action: click(start_box='='
(start_box='(318,285)')<|im_end|>
User: <|vision_start|><|image_pad|><|vision_end|><|im_end|>
Assistant:

[nivibilla]

another example::

Assistant: Thought: To proceed with finding the cheapest 4-star hotel in Paris for the first week of February, I need to adjust the dates in the "Dates" field. Clicking on the "Dates" field will allow me to modify the current selection and set the appropriate dates for the search.\nClick on the "Dates" field to begin modifying the search dates.\nAction: click(start_box=\'=\'\n(317,280))<|im_end|>\nUser: <|vision_start|><|image_pad|><|vision_end|><|im_end|>\nAssistant: Thought: To proceed with finding the cheapest 4-star hotel in Paris for the first week of February, I need to adjust the dates in the "Dates" field. Clicking on the "Dates" field will allow me to modify the current selection and set the appropriate dates for the search.\nClick on the "Dates" field to begin modifying the search dates.\nAction: click(start_box=\'=\'\n(317,280))<|im_end|>\nUser: <|vision_start|><|image_pad|><|vision_end|><|im_end|>\nAssistant:

The model is actually giving coordinates but i guess they are being parsed incorrectly

[nivibilla]