Some issues currently

[BatmanofZuhandArrgh]

Hi,
Just fyi:

• Currently in the config file, dynamic replanning is not enabled
• src/run_eval.py redefining the cur_task dict at line 118 is redundant and causing erroneous prompt
• Currently run_eval.py is not incorporating logit_bias, also had to did some work to get goal-condition success rate out of env.envs[0].get_info()
  Also have you experimented with temperature in text generation, to avoid loops where the agent assist on doing something wrong?

[BatmanofZuhandArrgh]

y'all are driving me crazy. THis is in run_eval.py

[lxsy-xcy]

Did you reproduce this codebase without any bugs？I'm trying to reproduce it, but get a lot "Nothing Happens"

[BatmanofZuhandArrgh]

@lxsy-xcy No i did a lot of modification before it works, and still has lots of bugs

[lxsy-xcy]

@lxsy-xcy No i did a lot of modification before it works, and still has lots of bugs

Sorry to hear that

[charlotteannchen]

I've successfully reproduced it, but I also received a lot of 'Nothing Happens' (almost all of the observations) while evaluating the 'eval_in_distribution' split. Is this normal? There are 0 plan in completed_plans
in each task.

[BatmanofZuhandArrgh]

@charlotteannchen I believe Nothing Happens is because of the natural language instruction output by the model is in the wrong format. In my installation of alfworld, it should look like "go to fridge 1", "take spoon 1" or whatever, iirc. You gotta search what the correct grammar is in your installation tho

[chanhee-luke]

Hi, there is an error with the underlying simulator (i.e. simulator can't locate the object). So we reverted the code to only generated high-level plans for now. We are working on a fix right now. In the meantime, I recommend a recently released codebase (https://github.com/lbaa2022/LLMTaskPlanning) that covers a similar functionality. Thanks for the interest!