Cannot reproduce result on ObstructedMaze-2Dlhb

[sungwoong]

Hi @tianjunz

I ran your codes with the same command you suggested as
OMP_NUM_THREADS=1 python main.py --model bebold --env MiniGrid-ObstructedMaze-2Dlhb-v0 --total_frames 500000000 --intrinsic_reward_coef 0.05 --entropy_cost 0.0005

However, the obtained "mean episode return" is still 0 even after 60M frames, which is different from that in Fig.4 in the paper (NovelD).
my log: logs.csv
Could you check it or share your result (log)?

FYI, MultiRoom and KeyCorridor tasks seem to be reproduced. I used the following versions: pytorch(1.10.0), gym(0.15.4), gym_minigrid(1.0.2).

Thanks,
Sungwoong.

[CrazySssst]

I meet the same issue. Do you solve it ? @sungwoong @tianjunz

[CrazySssst]

Here is my hyper-parameters

`args: {

    "alpha": 0.99,

    "baseline_cost": 0.5,

    "batch_size": 32,

    "checkpoint_num_frames": 10000000,

    "disable_checkpoint": false,

    "disable_cuda": false,

    "discounting": 0.99,

    "ema_momentum": 1.0,

    "entropy_cost": 0.0005,

    "env": "MiniGrid-ObstructedMaze-2Dlhb-v0",

    "env_seed": 1,

    "epsilon": 1e-05,

    "fix_seed": false,

    "forward_loss_coef": 10.0,

    "init_num_frames": 1000000.0,

    "intrinsic_reward_coef": 0.05,

    "inverse_loss_coef": 0.1,

    "learning_rate": 0.0001,

    "max_grad_norm": 40.0,

    "model": "bebold",

    "momentum": 0,

    "no_reward": false,

    "num_actors": 40,

    "num_buffers": 80,

    "num_input_frames": 1,

    "num_threads": 4,

    "planning_intrinsic_reward_coef": 0.5,

    "predictor_learning_rate": 0.0001,

    "queue_timeout": 1,

    "rnd_loss_coef": 1.0,

    "run_id": 0,

    "save_interval": 10,

    "savedir": "./experiments/",

    "scale_fac": 0.5,

    "seed": 0,

    "state_embedding_dim": 256,

    "target_update_freq": 2,

    "total_frames": 500000000,

    "unroll_length": 100,

    "use_fullobs_intrinsic": false,

    "use_fullobs_policy": false,

    "use_lstm": false,

    "use_lstm_intrinsic": false,

    "xpid": "MiniGrid-ObstructedMaze-2Dlhb-v0-bebold-20220429-135522"

}

`

[swan-utokyo]

Hi @sungwoong and @CrazySssst , I am writing this in case you are still interested in reproducing NovelD's results.

We re-implemented NovelD based on its original paper and the code released in this repository, and successfully reproduced reasonably good results in ObstructedMaze-Full (which is even more difficult than 2Dlhb).

For your reference, here are our paper and our implementation of NovelD. Hyperparameters used by us can be found in the appendix of our paper.