[E2E] Timm/Torchbench models got "fail_accuracy"

[libohao1201]

🐛 Describe the bug

The following models got fail_accuracy

		fail_accuracy
LNL	HF
	Timm
	Torchbench	hf_Roberta_base (train_eager_fp16/bf16) hf_BigBird (train_eager_amp_fp16/bf16) hf_Reformer (train_eager_amp_fp16/bf16) hf_Whisper (train_eager_amp_fp16/bf16)
BMG	HF
	Timm
	Torchbench	hf_Reformer (train_eager_amp_bf16/fp16) hf_Whisper (train_eager_amp_fp16/bf16)
ARC	HF
	Timm	coat_lite_mini(train_eager_fp32)
	Torchbench	hf_Whisper (train_eager_amp_bf16/fp16)

Versions

Env:

stock pytorch	https://github.com/pytorch/pytorch/tree/90b7dcf2c5ee13b892701822f2abbc0e64f5584d pip install --pre torch==2.6.0.dev20241202+xpu torchvision==0.20.0.dev20241202+xpu torchaudio==2.5.0.dev20241202+xpu --index-url https://download.pytorch.org/whl/nightly/xpu
torch-xpu-ops	Commit: bf4bab1 Commit: 0f48ac0 (including #1187) - for UT
Driver	32.0.101.6314 32.0.101.6252(bmg)
Conda	python 3.10
transformer	243e186efbf7fb93328dd6b34927a4e8c8f24395