The output logits are inconsistent when inputs are totally the same.

[fhdnskfbeuv]

Intriguing phenomenon occurred when I inputted the same object (downloaded here input) into Qwen2-VL-2B-Instruct twice.

import contextlib
import copy

import torch
import torch.nn.functional as F
from transformers import Qwen2VLForConditionalGeneration

if __name__ == '__main__':
    torch.use_deterministic_algorithms(True, warn_only=True)  # won't help
    torch.backends.cudnn.benchmark = False  # won't help
    torch.backends.cuda.matmul.allow_tf32 = True  # won't help
    torch.backends.cudnn.allow_tf32 = True  # won't help
    torch.backends.cudnn.deterministic = True  # won't help
    model_kwargs = dict(
        revision='main',
        torch_dtype=torch.float16,
    )
    model = Qwen2VLForConditionalGeneration.from_pretrained('Qwen/Qwen2-VL-2B-Instruct',
                                                            **model_kwargs, )
    model.eval()
    device = torch.device('cuda:7')
    ctx_manager = contextlib.nullcontext()
    compute_loss_context_manager = contextlib.nullcontext()
    batch = torch.load('./input.pt')
    batch = {k: v if not hasattr(v, 'to') else v.to(device) for k, v in batch.items()}
    model.to(device)
    with torch.no_grad():
        l1 = model(**copy.deepcopy(batch))
    with torch.no_grad():
        l2 = model(**copy.deepcopy(batch))
    print(F.cosine_similarity(l1.logits.float(), l2.logits.float(), dim=-1).mean())  # != 1.0
    print(F.mse_loss(l1.logits.float(), l2.logits.float()).mean())  # !=0.0

You may find that l1 and l2 are inconsistent. Is this a bug or a feature?

[Natyren]

Did you try to manual seed torch and torch.cuda?
like this

torch.manual_seed(args.seed)
torch.cuda.manual_seed_all(args.seed)